[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 498219019 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003135] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005012] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.011100] LSM: Security Framework initializing [ 0.012049] Yama: becoming mindful. [ 0.013034] SELinux: Initializing. [ 0.014056] *** VALIDATE selinux *** [ 0.020696] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024568] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025157] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026112] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027101] *** VALIDATE tmpfs *** [ 0.029245] *** VALIDATE proc *** [ 0.030245] *** VALIDATE cgroup *** [ 0.031012] *** VALIDATE cgroup2 *** [ 0.032278] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033157] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035028] Spectre V2 : User space: Vulnerable [ 0.036010] Speculative Store Bypass: Vulnerable [ 0.039364] debug: unmapping init [mem 0xffffffffb5659000-0xffffffffb5660fff] [ 0.041163] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042584] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043023] ... version: 2 [ 0.044017] ... bit width: 48 [ 0.045011] ... generic registers: 4 [ 0.046014] ... value mask: 0000ffffffffffff [ 0.047016] ... max period: 00007fffffffffff [ 0.048017] ... fixed-purpose events: 3 [ 0.049013] ... event mask: 000000070000000f [ 0.050290] rcu: Hierarchical SRCU implementation. [ 0.052429] smp: Bringing up secondary CPUs ... [ 0.053610] x86: Booting SMP configuration: [ 0.054031] .... node #0, CPUs: #1 #2 #3 [ 0.058210] smp: Brought up 1 node, 4 CPUs [ 0.060018] smpboot: Max logical packages: 1 [ 0.061009] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.138019] node 0 deferred pages initialised in 74ms [ 0.140310] devtmpfs: initialized [ 0.141230] x86/mm: Memory block size: 128MB [ 0.143872] gcov: version magic: 0x41383552 [ 0.144604] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.148170] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.151464] pinctrl core: initialized pinctrl subsystem [ 0.154249] [ 0.154874] ************************************************************* [ 0.157021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.160021] ** ** [ 0.162019] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.165020] ** ** [ 0.167019] ** This means that this kernel is built to expose internal ** [ 0.170021] ** IOMMU data structures, which may compromise security on ** [ 0.172021] ** your system. ** [ 0.175021] ** ** [ 0.178019] ** If you see this message and you are not debugging the ** [ 0.180016] ** kernel, report this immediately to your vendor! ** [ 0.183023] ** ** [ 0.186021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.188020] ************************************************************* [ 0.191891] NET: Registered protocol family 16 [ 0.193552] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.197079] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.200087] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.204081] cpuidle: using governor menu [ 0.205697] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.208475] PCI: Using configuration type 1 for base access [ 0.211131] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.222069] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.224033] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.227096] cryptd: max_cpu_qlen set to 1000 [ 0.231262] ACPI: Added _OSI(Module Device) [ 0.233017] ACPI: Added _OSI(Processor Device) [ 0.234011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.236016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.240848] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.246508] ACPI: Interpreter enabled [ 0.248063] ACPI: PM: (supports S0 S3 S4 S5) [ 0.250014] ACPI: Using IOAPIC for interrupt routing [ 0.251122] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.255399] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.266525] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.269071] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.272051] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.276113] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.282577] acpiphp: Slot [2] registered [ 0.284208] acpiphp: Slot [5] registered [ 0.285221] acpiphp: Slot [6] registered [ 0.287193] acpiphp: Slot [7] registered [ 0.289181] acpiphp: Slot [8] registered [ 0.291288] acpiphp: Slot [9] registered [ 0.293194] acpiphp: Slot [10] registered [ 0.294115] acpiphp: Slot [3] registered [ 0.296136] acpiphp: Slot [4] registered [ 0.297334] acpiphp: Slot [11] registered [ 0.299095] acpiphp: Slot [12] registered [ 0.300106] acpiphp: Slot [13] registered [ 0.302136] acpiphp: Slot [14] registered [ 0.303103] acpiphp: Slot [15] registered [ 0.305109] acpiphp: Slot [16] registered [ 0.306118] acpiphp: Slot [17] registered [ 0.308096] acpiphp: Slot [18] registered [ 0.309108] acpiphp: Slot [19] registered [ 0.311101] acpiphp: Slot [20] registered [ 0.312107] acpiphp: Slot [21] registered [ 0.314114] acpiphp: Slot [22] registered [ 0.315105] acpiphp: Slot [23] registered [ 0.316067] acpiphp: Slot [24] registered [ 0.318111] acpiphp: Slot [25] registered [ 0.319117] acpiphp: Slot [26] registered [ 0.321109] acpiphp: Slot [27] registered [ 0.322104] acpiphp: Slot [28] registered [ 0.323118] acpiphp: Slot [29] registered [ 0.325116] acpiphp: Slot [30] registered [ 0.326121] acpiphp: Slot [31] registered [ 0.328097] PCI host bridge to bus 0000:00 [ 0.330030] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.333038] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.335034] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.338033] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.340030] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.343041] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.345276] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.349248] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.353490] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.363745] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.369020] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.372029] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.374024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.376029] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.379525] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.381518] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.383034] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.385622] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.391017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.402926] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.408017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.412907] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.421022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.433024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.460023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.474961] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.483016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.494016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.509015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.521391] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.531021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.542021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.565021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.580405] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.591022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.599026] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.621024] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.632639] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.641024] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.649023] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.670019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.680930] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.689023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.696018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.717021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.731457] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.734457] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.737957] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.740457] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.742269] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.748221] iommu: Default domain type: Passthrough [ 0.750544] SCSI subsystem initialized [ 0.752226] ACPI: bus type USB registered [ 0.754128] usbcore: registered new interface driver usbfs [ 0.756066] usbcore: registered new interface driver hub [ 0.757080] usbcore: registered new device driver usb [ 0.759198] pps_core: LinuxPPS API ver. 1 registered [ 0.761013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.763061] PTP clock support registered [ 0.765183] EDAC MC: Ver: 3.0.0 [ 0.768189] PCI: Using ACPI for IRQ routing [ 0.769000] NetLabel: Initializing [ 0.770017] NetLabel: domain hash size = 128 [ 0.772020] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.775106] NetLabel: unlabeled traffic allowed by default [ 0.777224] vgaarb: loaded [ 0.779350] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.781021] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.788433] clocksource: Switched to clocksource kvm-clock [ 0.899392] VFS: Disk quotas dquot_6.6.0 [ 0.900845] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.903225] *** VALIDATE ramfs *** [ 0.904212] *** VALIDATE hugetlbfs *** [ 0.905947] pnp: PnP ACPI init [ 0.908168] pnp: PnP ACPI: found 6 devices [ 0.929546] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.933159] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.935412] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.937721] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.939860] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.942755] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.946926] NET: Registered protocol family 2 [ 0.949573] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.954609] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.958133] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.963942] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.967649] TCP: Hash tables configured (established 65536 bind 65536) [ 0.970738] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.974211] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.977218] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.979995] NET: Registered protocol family 1 [ 0.982356] RPC: Registered named UNIX socket transport module. [ 0.984583] RPC: Registered udp transport module. [ 0.986093] RPC: Registered tcp transport module. [ 0.987727] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.990053] NET: Registered protocol family 44 [ 0.991956] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.994439] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.996734] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.999405] PCI: CLS 0 bytes, default 64 [ 1.001997] Unpacking initramfs... [ 2.430886] debug: unmapping init [mem 0xffff94d57cc54000-0xffff94d57ffbffff] [ 2.437194] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.439606] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.444189] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.000296] Initialise system trusted keyrings [ 3.002443] Key type blacklist registered [ 3.007353] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.018382] zbud: loaded [ 3.022294] *** VALIDATE nfs *** [ 3.023398] *** VALIDATE nfs4 *** [ 3.025518] pstore: using deflate compression [ 3.028950] Platform Keyring initialized [ 3.132726] NET: Registered protocol family 38 [ 3.134736] Key type asymmetric registered [ 3.135905] Asymmetric key parser 'x509' registered [ 3.137605] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.140432] io scheduler mq-deadline registered [ 3.142530] io scheduler kyber registered [ 3.144376] io scheduler bfq registered [ 3.146319] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.148619] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.151287] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.154280] ACPI: Power Button [PWRF] [ 3.162447] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.169955] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.186030] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.195583] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.214209] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.241719] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.271336] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.276626] Non-volatile memory driver v1.3 [ 3.278899] Linux agpgart interface v0.103 [ 3.310518] virtio_blk virtio1: [vda] 150016 512-byte logical blocks (76.8 MB/73.3 MiB) [ 3.315458] vda: detected capacity change from 0 to 76808192 [ 3.331814] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.334208] vdb: detected capacity change from 0 to 1073741824 [ 3.350596] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.354021] vdc: detected capacity change from 0 to 2621440000 [ 3.369824] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.372584] vdd: detected capacity change from 0 to 2621440000 [ 3.393183] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.396606] vde: detected capacity change from 0 to 4294967296 [ 3.412378] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.415137] vdf: detected capacity change from 0 to 4294967296 [ 3.423248] libphy: Fixed MDIO Bus: probed [ 3.429697] usbcore: registered new interface driver usbserial_generic [ 3.432423] usbserial: USB Serial support registered for generic [ 3.434972] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.440106] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.442445] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.445433] mousedev: PS/2 mouse device common for all mice [ 3.448605] rtc_cmos 00:05: RTC can wake from S4 [ 3.451778] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.456025] rtc_cmos 00:05: registered as rtc0 [ 3.458500] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.458937] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.465130] intel_pstate: CPU model not supported [ 3.465141] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.470576] hid: raw HID events driver (C) Jiri Kosina [ 3.472764] usbcore: registered new interface driver usbhid [ 3.474662] usbhid: USB HID core driver [ 3.476271] drop_monitor: Initializing network drop monitor service [ 3.478662] Initializing XFRM netlink socket [ 3.480598] NET: Registered protocol family 10 [ 3.483715] Segment Routing with IPv6 [ 3.485182] NET: Registered protocol family 17 [ 3.487443] mpls_gso: MPLS GSO support [ 3.492787] RAS: Correctable Errors collector initialized. [ 3.495116] AVX version of gcm_enc/dec engaged. [ 3.496835] AES CTR mode by8 optimization enabled [ 3.572882] sched_clock: Marking stable (3572836479, 0)->(4433771610, -860935131) [ 3.577052] registered taskstats version 1 [ 3.579366] Loading compiled-in X.509 certificates [ 3.581342] zswap: loaded using pool lzo/zbud [ 3.609063] Key type big_key registered [ 3.622595] Key type encrypted registered [ 3.624286] ima: No TPM chip found, activating TPM-bypass! [ 3.626290] ima: Allocated hash algorithm: sha1 [ 3.628042] ima: No architecture policies found [ 3.630060] evm: Initialising EVM extended attributes: [ 3.632068] evm: security.selinux [ 3.633432] evm: security.ima [ 3.634637] evm: security.capability [ 3.636013] evm: HMAC attrs: 0x1 [ 3.638386] rtc_cmos 00:05: setting system clock to 2026-09-12 03:47:46 UTC (1789184866) [ 3.644288] debug: unmapping init [mem 0xffffffffb6603000-0xffffffffb67fffff] [ 3.647412] debug: unmapping init [mem 0xffffffffb5382000-0xffffffffb5658fff] [ 3.655225] Write protecting the kernel read-only data: 28672k [ 3.658639] debug: unmapping init [mem 0xffffffffb3a03000-0xffffffffb3bfffff] [ 3.661022] debug: unmapping init [mem 0xffffffffb4314000-0xffffffffb43fffff] [ 3.704254] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.712546] systemd[1]: Detected virtualization kvm. [ 3.713832] systemd[1]: Detected architecture x86-64. [ 3.715477] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.743900] systemd[1]: No hostname configured. [ 3.745706] systemd[1]: Set hostname to . [ 3.747703] random: systemd: uninitialized urandom read (16 bytes read) [ 3.750287] systemd[1]: Initializing machine ID from random generator. [ 3.810792] random: ln: uninitialized urandom read (6 bytes read) [ 3.900851] random: systemd: uninitialized urandom read (16 bytes read) [ 3.903978] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.910558] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ 3.916102] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Swap. Starting Journal Service... [ OK ] Listening on udev Kernel Socket. Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.099317] device-mapper: uevent: version 1.0.3 [ 5.103380] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 6.823772] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 6.888923] random: fast init done [ 7.076367] scsi host0: ata_piix [ 7.126637] scsi host1: ata_piix [ 7.129598] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.133697] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 12.149093] random: crng init done [ 12.155351] random: 7 urandom warning(s) missed due to ratelimiting [ 13.191966] dracut-initqueue[592]: RTNETLINK answers: File exists [ 13.543056] hrtimer: interrupt took 5962886 ns Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 14.872770] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 17.337024] printk: systemd: 26 output lines suppressed due to ratelimiting [ 18.207089] SELinux: Disabled at runtime. [ 18.332617] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.345865] systemd[1]: Detected virtualization kvm. [ 18.348609] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.956743] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.968407] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.987756] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 20.003750] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 20.012485] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 20.034741] systemd[1]: Starting Journal Service... Starting Journal Service... [ 20.047960] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. [ 20.568415] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 21.467878] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 22.374557] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 22.425080] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 22.837437] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 22.868572] EDAC sbridge: Ver: 1.1.2 [ 27.186452] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 27.977725] NFS: Registering the id_resolver key type [ 27.985720] Key type id_resolver registered [ 27.988569] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg645-server login: [ 78.510532] libcfs: loading out-of-tree module taints kernel. [ 78.559930] Key type ._llcrypt registered [ 78.562328] Key type .llcrypt registered [ 78.644226] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 97.010294] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 98.419854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 98.429425] alg: No test for adler32 (adler32-zlib) [ 99.908314] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 100.465318] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 102.255544] Key type lgssc registered [ 104.438952] Lustre: Echo OBD driver; http://www.lustre.org/ [ 121.959859] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 164.471731] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 175.091935] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 175.109235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 176.287895] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 176.327557] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 176.426431] Lustre: lustre-MDT0000: new disk, initializing [ 176.560183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 176.578110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 180.881163] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 193.133352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 193.222427] Lustre: 6517:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 193.283410] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 193.289676] Lustre: Skipped 1 previous similar message [ 193.399476] Lustre: lustre-MDT0001: new disk, initializing [ 193.464761] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 193.493929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 193.503867] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 197.427717] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 201.584084] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 210.676371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 211.003415] Lustre: lustre-OST0000: new disk, initializing [ 211.015530] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 211.021738] Lustre: 8455:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 211.092588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 215.572583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 215.587362] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 215.626641] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 217.718262] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 231.297201] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 231.426518] Lustre: lustre-OST0001: new disk, initializing [ 231.429549] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 231.434146] Lustre: 9527:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 231.496240] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 237.574130] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 241.181660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 241.190186] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 241.304178] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 248.609308] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 252.592909] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 257.829576] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing check_logdir /tmp/testlogs/ [ 262.544401] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing yml_node [ 266.477915] Lustre: DEBUG MARKER: Client: 2.17.58.40 [ 268.768356] Lustre: DEBUG MARKER: MDS: 2.17.58.40 [ 271.143061] Lustre: DEBUG MARKER: OSS: 2.17.58.40 [ 272.573947] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Sep 11 23:52:14 EDT 2026 [ 288.092954] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 289.535047] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 306.143714] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 306.149938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 306.165315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 307.680588] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 307.681691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 307.699402] Lustre: Skipped 1 previous similar message [ 307.707240] Lustre: Skipped 1 previous similar message [ 310.967283] Lustre: server umount lustre-MDT0000 complete [ 312.805744] LustreError: 6529:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 312.817278] LustreError: 6529:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 314.104372] LustreError: 6509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789185176 with bad export cookie 10997231218112460475 [ 314.108499] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 314.115203] LustreError: 6509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 317.923711] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 317.924597] LustreError: 6526:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 317.928874] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 317.928882] Lustre: Skipped 2 previous similar messages [ 317.935648] Lustre: Skipped 2 previous similar messages [ 317.968334] LustreError: 6526:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 319.967755] LustreError: 12061:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 319.969454] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 319.984406] LustreError: 12061:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 319.990235] Lustre: Skipped 2 previous similar messages [ 320.515911] Lustre: server umount lustre-MDT0001 complete [ 324.221725] Lustre: server umount lustre-OST0000 complete [ 327.572332] Lustre: server umount lustre-OST0001 complete [ 333.074657] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 340.972334] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 372.617512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 372.814626] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 372.858967] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 372.947219] Lustre: lustre-MDT0000: new disk, initializing [ 373.008811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 373.026828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 376.629568] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 385.085543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 385.245229] Lustre: 15078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 385.310888] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 385.314805] Lustre: Skipped 1 previous similar message [ 385.375576] Lustre: lustre-MDT0001: new disk, initializing [ 385.489538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 385.536116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 385.552160] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 389.216288] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 397.253665] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 402.442537] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 408.656391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 408.993848] Lustre: lustre-OST0000: new disk, initializing [ 408.998617] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 409.006524] Lustre: 17047:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 409.107879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 410.756514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 410.776926] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 410.865982] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 414.357354] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 422.476827] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 426.460992] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 426.612724] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 430.507507] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 430.719988] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 436.707686] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 436.709263] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 436.720369] LustreError: Skipped 1 previous similar message [ 436.721720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 436.762086] Lustre: Skipped 1 previous similar message [ 438.556519] Lustre: server umount lustre-OST0000 complete [ 446.945179] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 446.951704] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 446.968458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 446.985979] Lustre: Skipped 1 previous similar message [ 446.987396] Lustre: Skipped 1 previous similar message [ 448.698509] Lustre: server umount lustre-MDT0000 complete [ 452.069799] LustreError: 15089:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 452.090365] LustreError: 15089:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 452.959369] LustreError: 15068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789185315 with bad export cookie 10997231218112462001 [ 452.963128] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 452.966883] LustreError: 15068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 453.246809] Lustre: server umount lustre-MDT0001 complete [ 459.768787] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 23:55:21 (1789185321) [ 480.555585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 481.173591] LustreError: 19922:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 481.275848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 485.615719] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 491.497478] LustreError: 19922:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 491.517586] LustreError: 19922:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 493.838027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 497.853037] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 504.735816] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 510.186629] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 512.047906] Lustre: Failing over lustre-MDT0000 [ 512.283665] Lustre: server umount lustre-MDT0000 complete [ 514.532898] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 514.545248] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 514.569995] LustreError: 20651:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 530.252270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 530.351604] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 534.098418] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 536.033442] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 536.037993] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 536.053334] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 541.596472] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 546.273856] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 546.276539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 546.289193] Lustre: Skipped 2 previous similar messages [ 546.289992] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 546.311314] Lustre: Skipped 2 previous similar messages [ 551.558639] Lustre: server umount lustre-MDT0000 complete [ 554.542977] LustreError: 19903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789185417 with bad export cookie 10997231218112463422 [ 554.544329] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 554.553075] LustreError: 19903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 554.804969] Lustre: server umount lustre-MDT0001 complete [ 562.097226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 562.477570] LustreError: 23258:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 562.492773] LustreError: 23258:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 562.549237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 562.554183] Lustre: Skipped 2 previous similar messages [ 566.092760] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 573.464413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 577.879101] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 584.080366] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 588.737386] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 590.424485] Lustre: Failing over lustre-MDT0000 [ 590.577874] Lustre: server umount lustre-MDT0000 complete [ 592.356868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 592.369878] Lustre: Skipped 1 previous similar message [ 607.101903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 607.196550] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 607.440379] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 610.768379] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 612.841317] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 612.844711] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 612.852907] Lustre: Skipped 1 previous similar message [ 612.878405] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 618.236300] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 623.073449] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 623.082150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 623.085670] Lustre: Skipped 2 previous similar messages [ 636.383535] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 636.651735] Lustre: server umount lustre-MDT0000 complete [ 639.459342] LustreError: 23259:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 639.480872] LustreError: 23259:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 639.964758] LustreError: 23240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789185502 with bad export cookie 10997231218112464360 [ 639.970025] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 639.975666] LustreError: 23240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 640.188674] Lustre: server umount lustre-MDT0001 complete [ 648.475712] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 23:58:30 (1789185510) [ 664.128613] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 672.562911] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 708.149549] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 717.658498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 717.855429] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 717.891436] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 717.949569] Lustre: lustre-MDT0000: new disk, initializing [ 718.016466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 718.024293] Lustre: Skipped 2 previous similar messages [ 718.042143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 721.978781] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 731.008678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 731.096792] Lustre: 29630:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 731.332237] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 735.076383] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 742.957481] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 747.901694] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 753.786890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 753.993568] Lustre: 31601:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 755.956449] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 755.993868] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 756.051412] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 756.068204] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 758.960357] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 766.754444] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 773.893288] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 792.055308] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 792.055747] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 792.068364] Lustre: Skipped 2 previous similar messages [ 792.070427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 792.079318] LustreError: Skipped 1 previous similar message [ 792.101489] Lustre: Skipped 7 previous similar messages [ 796.696256] Lustre: server umount lustre-OST0000 complete [ 806.693981] Lustre: server umount lustre-MDT0000 complete [ 807.395615] LustreError: 29635:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 807.420556] LustreError: 29635:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 810.618356] LustreError: 29620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789185673 with bad export cookie 10997231218112464766 [ 810.626630] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 810.639063] LustreError: 29620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 811.161606] Lustre: server umount lustre-MDT0001 complete [ 823.280973] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 826.379866] Key type lgssc unregistered [ 826.808377] LNet: 33593:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 826.826667] LNetError: 33593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 826.850703] LNet: Removed LNI 192.168.206.145@tcp [ 827.906546] Key type .llcrypt unregistered [ 827.909822] Key type ._llcrypt unregistered [ 850.129606] Key type ._llcrypt registered [ 850.135638] Key type .llcrypt registered [ 850.257729] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 851.060128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 851.075631] alg: No test for adler32 (adler32-zlib) [ 852.130313] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 852.351961] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 854.031404] Key type lgssc registered [ 855.080906] Lustre: Echo OBD driver; http://www.lustre.org/ [ 861.366970] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 861.390752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 862.819247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 866.087206] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 872.423227] LustreError: 35002:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 872.439667] LustreError: 35002:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 872.493768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 872.840265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 876.103815] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 881.897776] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 886.058748] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 892.407235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 892.807206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 897.980965] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 900.401233] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 910.422801] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 925.018826] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 00:03:06 (1789185786) [ 926.868434] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 928.556504] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 936.306625] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 00:03:17 (1789185797) [ 945.391291] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 00:03:26 (1789185806) [ 946.594109] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 955.182187] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 955.192641] Lustre: Skipped 2 previous similar messages [ 956.601204] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 00:03:38 (1789185818) [ 959.801993] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 959.813442] Lustre: Skipped 3 previous similar messages [ 975.552679] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 975.559609] Lustre: Skipped 6 previous similar messages [ 977.938570] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 00:03:59 (1789185839) [ 980.297618] Lustre: *** cfs_fail_loc=131b, val=2*** [ 983.399144] Lustre: *** cfs_fail_loc=131b, val=2*** [ 990.666566] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 00:04:12 (1789185852) [ 992.235968] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 992.240866] Lustre: Skipped 3 previous similar messages [ 1002.396957] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 00:04:24 (1789185864) [ 1014.756821] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1014.759292] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1014.760617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1014.776414] Lustre: Skipped 1 previous similar message [ 1019.873722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1019.880335] Lustre: Skipped 2 previous similar messages [ 1025.014421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1025.026152] Lustre: Skipped 1 previous similar message [ 1028.063214] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1028.175369] Lustre: server umount lustre-OST0000 complete [ 1031.648793] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1031.654977] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1031.662751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1035.235530] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1037.197532] Lustre: server umount lustre-MDT0000 complete [ 1040.348341] LustreError: 34983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789185903 with bad export cookie 3420936810571851669 [ 1040.364110] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1040.366334] LustreError: 34983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1040.624882] Lustre: server umount lustre-MDT0001 complete [ 1048.297168] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 1050.363345] Key type lgssc unregistered [ 1050.581973] LNet: 42855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1050.594244] LNetError: 42855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1050.608682] LNet: Removed LNI 192.168.206.145@tcp [ 1051.241464] Key type .llcrypt unregistered [ 1051.244132] Key type ._llcrypt unregistered [ 1065.568913] Key type ._llcrypt registered [ 1065.573853] Key type .llcrypt registered [ 1065.668081] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 1077.003262] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 1077.609766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1077.661356] alg: No test for adler32 (adler32-zlib) [ 1078.619509] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 1078.842394] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 1080.495187] Key type lgssc registered [ 1081.373798] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1108.236624] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1108.252611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1109.431261] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1109.462683] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1109.522204] Lustre: lustre-MDT0000: new disk, initializing [ 1109.568609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1109.582678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1112.587456] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1121.440812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1121.534804] Lustre: 45981:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1121.586139] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1121.591962] Lustre: Skipped 1 previous similar message [ 1121.651957] Lustre: lustre-MDT0001: new disk, initializing [ 1121.735451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1121.779075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1121.793037] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1125.243318] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1132.612503] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1137.394607] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1142.946267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1143.184951] Lustre: lustre-OST0000: new disk, initializing [ 1143.188494] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1143.194120] Lustre: 47953:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1143.274811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1148.298702] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1148.997439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1149.005293] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1149.124026] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1156.207432] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1160.172779] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1160.326338] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1163.717526] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1163.874155] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1169.892926] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1169.897832] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1169.905387] LustreError: Skipped 1 previous similar message [ 1169.907533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1169.918624] Lustre: Skipped 1 previous similar message [ 1169.935444] Lustre: Skipped 1 previous similar message [ 1171.559948] Lustre: server umount lustre-OST0000 complete [ 1175.009570] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1175.015160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1175.023933] Lustre: Skipped 1 previous similar message [ 1180.130544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1180.137086] Lustre: Skipped 2 previous similar messages [ 1180.849953] Lustre: server umount lustre-MDT0000 complete [ 1183.289443] LustreError: 46780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789186046 with bad export cookie 10908264830344025625 [ 1183.292522] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1183.296331] LustreError: 46780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1183.546768] Lustre: server umount lustre-MDT0001 complete [ 1190.542146] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 00:07:32 (1789186052) [ 1195.709300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1196.174177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1199.593349] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1201.638796] LustreError: 50411:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1201.649141] LustreError: 50411:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1205.669674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1205.732432] LustreError: 50410:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1205.954887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1209.666892] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1215.959786] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1220.390059] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1225.390322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1225.630880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1230.063559] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1235.288349] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1248.283813] LustreError: 52959:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1248.842987] Lustre: Modifying parameter general.timeout=1 in log params [ 1249.911947] Lustre: Modifying parameter general.timeout=3 in log params [ 1249.915367] Lustre: Skipped 1 previous similar message [ 1252.421424] Lustre: Modifying parameter general.timeout=7 in log params [ 1252.426789] Lustre: Skipped 3 previous similar messages [ 1256.805449] Lustre: Modifying parameter general.timeout=13 in log params [ 1256.812917] Lustre: Skipped 5 previous similar messages [ 1265.291410] Lustre: Modifying parameter general.timeout=25 in log params [ 1265.298132] Lustre: Skipped 11 previous similar messages [ 1281.371312] Lustre: Modifying parameter general.timeout=50 in log params [ 1281.375638] Lustre: Skipped 24 previous similar messages [ 1314.055244] Lustre: Modifying parameter general.timeout=99 in log params [ 1314.058651] Lustre: Skipped 48 previous similar messages [ 1346.757525] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 00:10:08 (1789186208) [ 1348.947327] Lustre: 59882:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1350.212692] Lustre: 59981:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1352.731304] Lustre: 60194:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1352.739930] Lustre: 60194:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1375.872347] Lustre: 61737:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1375.879620] Lustre: 61737:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1375.892360] LustreError: 61737:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1378.780355] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 00:10:40 (1789186240) [ 1381.388135] Lustre: Setting parameter general.jobid_var=TESTNAME in log params [ 1381.391383] Lustre: Skipped 52 previous similar messages [ 1390.560328] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1390.566200] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1390.576310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1392.611430] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1393.837856] Lustre: server umount lustre-MDT0000 complete [ 1396.384163] LustreError: 52251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789186259 with bad export cookie 10908264830344026500 [ 1396.388551] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1396.394461] LustreError: 52251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1396.599291] Lustre: server umount lustre-MDT0001 complete [ 1409.901536] Lustre: server umount lustre-OST0000 complete [ 1426.878332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1427.091091] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1427.242743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1429.880960] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1435.269105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1435.365817] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1435.393099] Lustre: 64266:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1438.914957] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1443.720700] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1447.453629] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1453.010159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1453.126995] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1458.082220] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1463.233803] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1464.680448] Lustre: 66834:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1472.993082] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1473.004592] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1473.017468] Lustre: Skipped 1 previous similar message [ 1473.022519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1473.028779] Lustre: Skipped 4 previous similar messages [ 1476.259654] Lustre: server umount lustre-OST0000 complete [ 1481.699333] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1481.707187] LustreError: Skipped 1 previous similar message [ 1481.711898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1481.721462] Lustre: Skipped 1 previous similar message [ 1481.729895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1487.881919] Lustre: server umount lustre-MDT0000 complete [ 1488.363315] LustreError: 65092:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1488.379987] LustreError: 65092:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1490.697735] LustreError: 66129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789186353 with bad export cookie 10908264830344029258 [ 1490.707226] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1490.707868] LustreError: 66129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1490.900842] Lustre: server umount lustre-MDT0001 complete [ 1499.072242] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 1500.903995] Key type lgssc unregistered [ 1501.118568] LNet: 68039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1501.122759] LNetError: 68039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1501.139927] LNet: Removed LNI 192.168.206.145@tcp [ 1501.714929] Key type .llcrypt unregistered [ 1501.718936] Key type ._llcrypt unregistered [ 1519.016232] Key type ._llcrypt registered [ 1519.018138] Key type .llcrypt registered [ 1519.115879] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 1519.800967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1519.812048] alg: No test for adler32 (adler32-zlib) [ 1520.805291] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 1521.031310] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 1522.663299] Key type lgssc registered [ 1523.425260] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1529.172853] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1529.208142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1530.578558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1534.089607] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1540.977335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1541.088235] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1541.383376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1544.745390] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1550.389985] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1554.534670] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1559.478370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1559.956362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1563.923784] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1569.331917] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1580.168605] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1587.170705] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1587.175281] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1587.184465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1590.756381] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1590.756599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1590.769146] Lustre: Skipped 1 previous similar message [ 1590.775180] Lustre: Skipped 2 previous similar messages [ 1592.299577] Lustre: server umount lustre-MDT0000 complete [ 1595.272108] LustreError: 69405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789186458 with bad export cookie 2459703211533383098 [ 1595.282385] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1595.286485] LustreError: 69405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1595.570858] Lustre: server umount lustre-MDT0001 complete [ 1609.147739] Lustre: server umount lustre-OST0000 complete [ 1630.237924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1630.430420] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1630.462210] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1630.673349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1633.891716] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1640.911927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1641.013909] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1641.036682] Lustre: 74291:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1641.510387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1645.088995] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1651.331917] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1655.998477] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1660.748046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1660.876126] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1661.133907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1665.312269] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1670.123571] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1671.328688] Lustre: 76874:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1672.578825] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1678.304427] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1678.316231] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1678.329370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1681.741659] Lustre: server umount lustre-OST0000 complete [ 1687.520592] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1687.526193] LustreError: Skipped 1 previous similar message [ 1687.528570] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1687.535360] Lustre: Skipped 1 previous similar message [ 1687.538173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1687.545061] Lustre: Skipped 1 previous similar message [ 1690.413489] Lustre: server umount lustre-MDT0000 complete [ 1692.740186] LustreError: 74284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789186555 with bad export cookie 2459703211533383938 [ 1692.742890] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1692.749670] LustreError: 74284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1692.937698] Lustre: server umount lustre-MDT0001 complete [ 1699.820594] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 1702.351484] Key type lgssc unregistered [ 1702.615591] LNet: 78083:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1702.621729] LNetError: 78083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1702.638373] LNet: Removed LNI 192.168.206.145@tcp [ 1703.347060] Key type .llcrypt unregistered [ 1703.349323] Key type ._llcrypt unregistered [ 1721.719352] Key type ._llcrypt registered [ 1721.721548] Key type .llcrypt registered [ 1721.808229] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 1722.442978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1722.555798] alg: No test for adler32 (adler32-zlib) [ 1723.590180] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 1723.804810] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 1725.487271] Key type lgssc registered [ 1726.396504] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1732.881679] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1732.907774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1734.364313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1737.785878] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1744.388665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1744.931495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1748.335489] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1753.768635] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1757.922160] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1763.068696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1763.713396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1768.029218] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1773.335306] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1787.836266] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1788.457576] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1789.793413] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1789.799530] Lustre: Skipped 1 previous similar message [ 1791.848535] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1791.859149] Lustre: Skipped 2 previous similar messages [ 1796.407340] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1796.415832] Lustre: Skipped 6 previous similar messages [ 1804.560502] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1804.564559] Lustre: Skipped 11 previous similar messages [ 1820.992286] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1820.995013] Lustre: Skipped 26 previous similar messages [ 1853.247638] Lustre: 87576:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1853.769838] Lustre: Disabling parameter general.jobid_var= in log params [ 1853.772938] Lustre: Skipped 47 previous similar messages [ 1854.852080] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 00:18:37 (1789186717) [ 1855.845989] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1857.069783] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 00:18:39 (1789186719) [ 1862.113818] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1862.120886] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1862.134616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1863.144146] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1863.144563] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1863.145654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1865.874632] Lustre: server umount lustre-OST0000 complete [ 1872.868228] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1872.873667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1872.888506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1872.893567] Lustre: Skipped 1 previous similar message [ 1874.713809] Lustre: server umount lustre-MDT0000 complete [ 1877.313187] LustreError: 82081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789186740 with bad export cookie 4937161105065659552 [ 1877.321800] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1877.322678] LustreError: 82081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1877.564657] Lustre: server umount lustre-MDT0001 complete [ 1884.913235] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 1886.804396] Key type lgssc unregistered [ 1887.019475] LNet: 88821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1887.025779] LNetError: 88821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1887.038257] LNet: Removed LNI 192.168.206.145@tcp [ 1887.602073] Key type .llcrypt unregistered [ 1887.603897] Key type ._llcrypt unregistered [ 1890.846373] Key type ._llcrypt registered [ 1890.847804] Key type .llcrypt registered [ 1890.912929] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1894.415106] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules [ 1895.118655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1895.139112] alg: No test for adler32 (adler32-zlib) [ 1896.078626] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 1896.227433] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 1897.882133] Key type lgssc registered [ 1898.561951] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1899.342205] LustreError: 90019:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 1910.969341] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 1916.865277] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1921.028443] LustreError: 90019:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1921.046265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1922.376707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1925.019290] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1929.772941] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 00:19:52 (1789186792) [ 1932.378621] LustreError: 91073:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1932.385180] LustreError: 91073:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1932.388627] LustreError: 91073:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -5 [ 1932.563547] Lustre: server umount lustre-MDT0000 complete [ 1939.983127] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 1941.456188] Key type lgssc unregistered [ 1941.636720] LNet: 92565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1941.642154] LNetError: 92565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1941.653220] LNet: Removed LNI 192.168.206.145@tcp [ 1942.160185] Key type .llcrypt unregistered [ 1942.164389] Key type ._llcrypt unregistered [ 1952.642500] Key type ._llcrypt registered [ 1952.644894] Key type .llcrypt registered [ 1952.708572] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 1953.220298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1953.262475] alg: No test for adler32 (adler32-zlib) [ 1954.233456] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 1954.400857] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 1956.031189] Key type lgssc registered [ 1956.773372] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1961.166993] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1961.183801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1962.494432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1964.924984] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1969.699173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1970.103501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1972.333232] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1976.286554] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1979.038607] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1982.476583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1982.771195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1985.366449] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1989.158762] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2023.838277] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 00:21:26 (1789186886) [ 2024.722257] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2025.661082] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 00:21:28 (1789186888) [ 2031.074885] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2031.078122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2031.079373] Lustre: Skipped 1 previous similar message [ 2033.619472] Lustre: server umount lustre-MDT0000 complete [ 2035.826838] LustreError: 93739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789186898 with bad export cookie 17572922237019620793 [ 2035.831843] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2035.839833] LustreError: 93739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2036.012077] Lustre: server umount lustre-MDT0001 complete [ 2048.918946] Lustre: server umount lustre-OST0000 complete [ 2056.496880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2056.884577] LustreError: 98027:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2056.900371] LustreError: 98027:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2057.046094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2059.839238] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2062.308836] LustreError: 98027:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2062.323216] LustreError: 98027:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2065.274179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2065.729360] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2068.505239] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2072.988610] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2076.044800] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2083.333067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2083.464896] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2083.470155] LustreError: 98015:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2083.484449] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2083.488801] LustreError: 100111:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2083.493707] LustreError: 100111:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2083.498648] LustreError: 100111:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2083.538409] Lustre: server umount lustre-OST0000 complete [ 2083.540779] LustreError: 100111:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2087.811906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2087.949952] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2087.956987] LustreError: 98015:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2087.963252] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2087.971273] LustreError: 100573:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2087.976410] LustreError: 100573:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2087.981660] LustreError: 100573:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2088.019573] Lustre: server umount lustre-OST0000 complete [ 2088.021471] LustreError: 100573:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2093.580786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2093.718599] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2093.722982] Lustre: Found index 0 for lustre-OST0000, updating log [ 2093.734147] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2093.819644] Lustre: lustre-OST0000: new disk, initializing [ 2093.822735] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2093.828115] Lustre: 101109:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2094.028388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2098.036384] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2100.335242] Lustre: Failing over lustre-OST0000 [ 2100.443939] Lustre: server umount lustre-OST0000 complete [ 2106.340764] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2106.347465] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2106.358116] Lustre: Skipped 1 previous similar message [ 2106.362640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2106.368298] Lustre: Skipped 2 previous similar messages [ 2110.944993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2110.947991] Lustre: Skipped 2 previous similar messages [ 2116.064937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2116.070178] Lustre: Skipped 1 previous similar message [ 2117.087414] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2117.192582] Lustre: server umount lustre-MDT0000 complete [ 2119.533312] LustreError: 98009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789186982 with bad export cookie 17572922237019623530 [ 2119.537771] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2119.541956] LustreError: 98009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2119.761932] Lustre: server umount lustre-MDT0001 complete [ 2125.184886] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 00:23:07 (1789186987) [ 2135.193940] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 2140.825863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2141.019506] LustreError: 103451:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2141.069966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2143.135102] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2146.273504] LustreError: 103452:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2147.913629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2150.194820] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2151.554194] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2154.704707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2158.330760] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2162.971706] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2163.017634] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2163.068034] Lustre: lustre-OST0001: new disk, initializing [ 2163.070996] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2163.075514] Lustre: 105769:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2164.896937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2164.902458] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2164.960936] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2166.210245] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2170.458684] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2177.832929] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2181.088287] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2181.092944] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2181.100484] Lustre: Skipped 1 previous similar message [ 2181.103877] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2191.841525] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2191.847042] Lustre: Skipped 9 previous similar messages [ 2194.399224] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2194.500641] Lustre: server umount lustre-MDT0001 complete [ 2196.961935] LustreError: 103451:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2196.972023] LustreError: 103451:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2199.933502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2200.000427] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2200.005962] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2200.046515] Lustre: 103440:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2200.508698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2200.512444] Lustre: Skipped 3 previous similar messages [ 2201.195462] LustreError: 107348:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2201.201867] LustreError: 107348:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2201.207900] LustreError: 103443:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2201.260755] LustreError: 107353:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.145@tcp: cfg command failed: rc = -17 [ 2201.267171] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2201.267171] [ 2203.119231] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2205.676117] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2208.981544] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 00:24:31 (1789187071) [ 2219.049520] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 2229.159914] Lustre: 109380:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2241.750564] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2243.599359] Lustre: 110513:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2302.944562] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2302.944854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2302.945025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2302.945031] Lustre: Skipped 1 previous similar message [ 2302.952751] Lustre: Skipped 4 previous similar messages [ 2307.522592] Lustre: server umount lustre-MDT0000 complete [ 2308.065232] LustreError: 103448:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2308.073784] LustreError: 103448:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2309.415680] LustreError: 103432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789187172 with bad export cookie 17572922237019624461 [ 2309.416083] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2309.420672] LustreError: 103432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2309.548798] Lustre: server umount lustre-MDT0001 complete [ 2321.426370] Lustre: server umount lustre-OST0000 complete [ 2333.487407] Lustre: server umount lustre-OST0001 complete [ 2336.213246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2341.014074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2344.948687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2349.690346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2358.415664] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 2362.955063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2362.969615] Lustre: lustre-MDT0000: reset Object Index mappings [ 2363.170663] LustreError: 114459:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2363.213583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2365.087550] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2368.831589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2368.842439] Lustre: lustre-MDT0001: reset Object Index mappings [ 2371.020542] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2372.340884] Lustre: 115608:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2375.603963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2378.545784] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2379.048435] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 2381.977819] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2384.671553] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2387.432094] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 2388.458905] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 2388.461349] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 2394.072233] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2395.747075] Lustre: 117484:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2420.630968] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 00:28:03 (1789187283) [ 2449.890461] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2449.894886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2449.900451] Lustre: Skipped 3 previous similar messages [ 2449.902949] Lustre: Skipped 4 previous similar messages [ 2455.152058] Lustre: server umount lustre-MDT0000 complete [ 2456.647053] LustreError: 114441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789187319 with bad export cookie 17572922237019671221 [ 2456.653666] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2456.654787] LustreError: 114441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2460.128553] LustreError: 114454:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2460.129305] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2460.136225] LustreError: 114454:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2460.142345] Lustre: Skipped 2 previous similar messages [ 2462.984387] Lustre: server umount lustre-MDT0001 complete [ 2470.675136] Lustre: server umount lustre-OST0000 complete [ 2478.524779] Lustre: server umount lustre-OST0001 complete [ 2481.354956] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 2485.742269] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 2503.456821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2503.574291] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2503.594650] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2503.642517] Lustre: lustre-MDT0000: new disk, initializing [ 2503.681717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2503.684374] Lustre: Skipped 3 previous similar messages [ 2503.691304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2505.708843] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2509.580829] Lustre: server umount lustre-MDT0000 complete [ 2513.621454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2513.772641] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2513.781572] Lustre: 121742:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2513.789389] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2515.770467] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2517.973541] Lustre: server umount lustre-MDT0000 complete [ 2522.279360] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 00:29:44 (1789187384) [ 2522.994615] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2523.793990] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 00:29:46 (1789187386) [ 2531.194489] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 2535.444078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2537.334090] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2540.798982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2540.848497] Lustre: 123792:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2540.882582] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2540.885385] Lustre: Skipped 1 previous similar message [ 2540.924996] Lustre: lustre-MDT0001: new disk, initializing [ 2540.979828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2540.985033] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2542.585917] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2545.161802] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2548.089897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2548.209375] Lustre: lustre-OST0000: new disk, initializing [ 2548.211929] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2548.215483] Lustre: 125370:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2548.944521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2548.949500] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2548.988746] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2550.778529] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2555.807766] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2555.888993] Lustre: 126239:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2557.327595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2557.331580] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2557.375638] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2558.464199] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2563.622407] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2565.160181] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2726.989358] Lustre: 132626:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 2726.993585] Lustre: 132626:0:(osd_handler.c:2092:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 2726.996415] Lustre: 132626:0:(osd_handler.c:2099:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 2726.999411] Lustre: 132626:0:(osd_handler.c:2109:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 2727.002825] Lustre: 132626:0:(osd_handler.c:2116:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 2727.006299] Lustre: 132626:0:(osd_handler.c:2123:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2852.856302] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 00:35:15 (1789187715) [ 2864.611465] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2864.612822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2864.617223] Lustre: Skipped 1 previous similar message [ 2864.619888] Lustre: Skipped 10 previous similar messages [ 2870.936911] Lustre: server umount lustre-MDT0000 complete [ 2872.575875] LustreError: 132654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789187735 with bad export cookie 17572922237019730273 [ 2872.579862] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2872.580494] LustreError: 132654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2874.847961] LustreError: 127038:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2874.853987] LustreError: 127038:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2878.818489] Lustre: server umount lustre-MDT0001 complete [ 2886.401908] Lustre: server umount lustre-OST0000 complete [ 2893.920067] Lustre: server umount lustre-OST0001 complete [ 2896.434502] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 2900.227001] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 2915.079096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2915.190971] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2915.207048] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2915.209405] Lustre: Skipped 1 previous similar message [ 2915.250481] Lustre: lustre-MDT0000: new disk, initializing [ 2915.252550] Lustre: Skipped 1 previous similar message [ 2915.295596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2915.297867] Lustre: Skipped 5 previous similar messages [ 2915.306367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2917.092469] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2922.251280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2922.314828] Lustre: 135641:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2922.455463] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2924.118585] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2928.458254] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2930.685866] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2933.599884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2933.737444] Lustre: lustre-OST0000: new disk, initializing [ 2933.739576] Lustre: Skipped 1 previous similar message [ 2933.742116] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2933.745178] Lustre: Skipped 2 previous similar messages [ 2933.748466] Lustre: 137607:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2935.307477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2935.313968] Lustre: Skipped 1 previous similar message [ 2935.316870] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2935.362126] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2936.282455] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2940.729467] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2944.731471] Lustre: lustre-MDD0000: changelog on [ 2945.880879] Lustre: lustre-MDD0001: changelog on [ 3526.216471] Lustre: 135647:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 3557.357409] Lustre: lustre-MDD0001: changelog off [ 3558.446327] Lustre: lustre-MDD0000: changelog off [ 3564.512475] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3564.515221] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3564.518924] Lustre: Skipped 4 previous similar messages [ 3564.520577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3564.522200] Lustre: Skipped 11 previous similar messages [ 3565.024738] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3574.239121] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3574.307831] Lustre: server umount lustre-OST0000 complete [ 3577.823782] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3577.826157] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3577.829617] Lustre: Skipped 1 previous similar message [ 3581.753174] Lustre: server umount lustre-MDT0000 complete [ 3583.016712] LustreError: 135634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789188445 with bad export cookie 17572922237020533173 [ 3583.019766] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3583.022082] LustreError: 135634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3583.146280] Lustre: server umount lustre-MDT0001 complete [ 3586.630957] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 3587.621986] Key type lgssc unregistered [ 3587.743344] LNet: 146642:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3587.745473] LNetError: 146642:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3587.758275] LNet: Removed LNI 192.168.206.145@tcp [ 3588.064093] Key type .llcrypt unregistered [ 3588.065172] Key type ._llcrypt unregistered [ 3594.352365] Key type ._llcrypt registered [ 3594.353275] Key type .llcrypt registered [ 3594.396267] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 3600.368659] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 3600.652862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3600.693158] alg: No test for adler32 (adler32-zlib) [ 3601.546606] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 3601.637291] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 3603.215311] Key type lgssc registered [ 3603.545812] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3614.341116] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3614.349848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3615.430817] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3615.443109] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3615.483703] Lustre: lustre-MDT0000: new disk, initializing [ 3615.506363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3615.513231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3616.909762] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 3620.954697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3620.993626] Lustre: 149809:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3621.016157] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3621.018302] Lustre: Skipped 1 previous similar message [ 3621.056648] Lustre: lustre-MDT0001: new disk, initializing [ 3621.087818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3621.099607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3621.106050] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3622.374273] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 3625.924293] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3627.683521] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3629.957901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3630.061078] Lustre: lustre-OST0000: new disk, initializing [ 3630.062890] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3630.065519] Lustre: 151779:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3630.094026] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3630.628818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3630.632778] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3630.663885] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3632.082694] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 3636.097728] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3639.521507] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_module obdecho/obdecho [ 3640.170535] LustreError: 152782:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 3640.174762] LustreError: 152782:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 3640.818558] LustreError: 152878:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 3645.922295] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3645.924066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3645.925463] Lustre: Skipped 2 previous similar messages [ 3645.927313] Lustre: Skipped 1 previous similar message [ 3650.332541] Lustre: server umount lustre-MDT0000 complete [ 3651.625117] LustreError: 149802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789188514 with bad export cookie 12234567378935771707 [ 3651.628077] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3651.631716] LustreError: 149802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3651.737377] Lustre: server umount lustre-MDT0001 complete [ 3662.230358] Lustre: server umount lustre-OST0000 complete [ 3668.776261] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 3671.754139] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 3683.055162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3683.151081] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3683.160590] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3683.195884] Lustre: lustre-MDT0000: new disk, initializing [ 3683.214044] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3683.219045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3684.516585] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 3688.658562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3688.703896] Lustre: 156586:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3688.727430] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3688.729274] Lustre: Skipped 1 previous similar message [ 3688.765593] Lustre: lustre-MDT0001: new disk, initializing [ 3688.803482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3688.814400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3688.817637] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3690.193812] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 3694.008427] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3695.941636] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3698.435870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3698.546544] Lustre: lustre-OST0000: new disk, initializing [ 3698.548143] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3698.551198] Lustre: 158555:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3698.579297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3700.549193] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 3701.365241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3701.369601] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3701.407819] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3704.105811] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3705.454557] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3705.516429] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3706.810352] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3706.894211] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3711.970256] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3711.970663] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3711.972836] LustreError: Skipped 1 previous similar message [ 3711.973406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3711.977084] Lustre: Skipped 1 previous similar message [ 3711.979377] Lustre: Skipped 1 previous similar message [ 3713.613626] Lustre: server umount lustre-OST0000 complete [ 3717.088713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3717.089326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3717.093092] Lustre: Skipped 1 previous similar message [ 3717.096009] Lustre: Skipped 2 previous similar messages [ 3722.208043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3722.210122] Lustre: Skipped 1 previous similar message [ 3727.328579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3727.330312] Lustre: Skipped 1 previous similar message [ 3728.863164] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3728.926369] Lustre: server umount lustre-MDT0000 complete [ 3730.180190] LustreError: 156579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789188593 with bad export cookie 12234567378935772652 [ 3730.183101] LustreError: 156579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3730.183448] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3730.291888] Lustre: server umount lustre-MDT0001 complete [ 3731.490914] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 00:49:54 (1789188594) [ 3732.085532] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 3732.676965] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 00:49:55 (1789188595) [ 3735.327812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3735.478390] LustreError: 160963:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3735.485370] LustreError: 160963:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 3735.502620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3736.883043] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 3739.751277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3741.231027] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 3743.763488] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3745.505218] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3747.659547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3749.657663] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 3752.008668] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3765.727788] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3765.730106] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3765.734894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3770.786496] Lustre: server umount lustre-MDT0000 complete [ 3772.052057] LustreError: 160943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789188634 with bad export cookie 12234567378935773520 [ 3772.055318] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3772.055641] LustreError: 160943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3772.383813] LustreError: 160958:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3772.384119] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3772.395214] Lustre: Skipped 2 previous similar messages [ 3777.503904] LustreError: 160960:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3777.509282] LustreError: 160960:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3778.234849] Lustre: server umount lustre-MDT0001 complete [ 3780.400910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3784.315416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3784.382580] LustreError: 164821:0:(llog.c:1655:llog_backup()) MGC192.168.206.145@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 3784.384972] Lustre: 164821:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.145@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 3802.080065] LustreError: 164826:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3802.085479] LustreError: 164826:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3806.175950] LustreError: 164827:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3806.182188] LustreError: 164827:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3816.415878] LustreError: 164827:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3816.420412] LustreError: 164827:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 3834.445475] LustreError: 164828:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3834.451095] LustreError: 164828:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 3845.215213] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 3845.217066] Lustre: 164821:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.145@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 3867.104284] LustreError: 164857:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3867.108951] LustreError: 164857:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 21 previous similar messages [ 3886.054351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3886.057424] Lustre: Skipped 2 previous similar messages [ 3887.284692] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 3889.915683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3891.170744] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3891.172406] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xa9c9ea35aa0c9278 [ 3892.507955] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 3894.870618] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3895.421331] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3896.296554] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3896.299472] Lustre: Skipped 1 previous similar message [ 3897.261255] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3900.863783] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3906.528910] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3906.530847] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3906.531040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3906.531044] Lustre: Skipped 8 previous similar messages [ 3906.534151] Lustre: Skipped 1 previous similar message [ 3907.868530] Lustre: server umount lustre-MDT0000 complete [ 3909.142572] LustreError: 162804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789188772 with bad export cookie 12234567378935779960 [ 3909.146079] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3909.147456] LustreError: 162804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3909.265077] Lustre: server umount lustre-MDT0001 complete [ 3920.855159] Lustre: server umount lustre-OST0000 complete [ 3927.368336] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 3930.574688] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 3941.996481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3942.069559] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3942.078361] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3942.111419] Lustre: lustre-MDT0000: new disk, initializing [ 3942.136129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3943.421783] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 3947.413576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3947.452148] Lustre: 169893:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3947.544293] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3948.783472] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 3952.286780] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3953.984823] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3956.060810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3956.152606] Lustre: 171861:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3956.175688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3956.177311] Lustre: Skipped 3 previous similar messages [ 3956.634877] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3956.665556] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3957.938178] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 3961.422590] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3962.716562] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3962.777981] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3963.991172] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3964.053433] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3966.944231] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3966.946148] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3966.946315] LustreError: Skipped 1 previous similar message [ 3966.949578] Lustre: Skipped 1 previous similar message [ 3966.950328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3966.954104] Lustre: Skipped 3 previous similar messages [ 3970.701903] Lustre: server umount lustre-OST0000 complete [ 3986.399054] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3986.468864] Lustre: server umount lustre-MDT0000 complete [ 3987.742334] LustreError: 169884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789188850 with bad export cookie 12234567378935796753 [ 3987.745714] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3987.746440] LustreError: 169884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3987.857941] Lustre: server umount lustre-MDT0001 complete [ 3990.917361] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 00:54:13 (1789188853) [ 3993.265139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3993.422714] LustreError: 174270:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3993.428505] LustreError: 174270:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 17 previous similar messages [ 3994.877334] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 3997.810722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3999.435661] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4002.019102] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4003.973889] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4006.259445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4008.376893] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4010.778555] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4025.411240] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 00:54:48 (1789188888) [ 4027.871778] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4027.874082] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4027.877475] Lustre: Skipped 3 previous similar messages [ 4032.526336] Lustre: server umount lustre-OST0000 complete [ 4034.016335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4034.019190] Lustre: Skipped 11 previous similar messages [ 4048.351137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4048.409019] Lustre: server umount lustre-MDT0000 complete [ 4049.667195] LustreError: 174253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789188912 with bad export cookie 12234567378935797621 [ 4049.671097] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4049.671808] LustreError: 174253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4049.780740] Lustre: server umount lustre-MDT0001 complete [ 4053.485338] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 4054.491736] Key type lgssc unregistered [ 4054.623343] LNet: 178017:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4054.625825] LNetError: 178017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4054.637350] LNet: Removed LNI 192.168.206.145@tcp [ 4054.955169] Key type .llcrypt unregistered [ 4054.956225] Key type ._llcrypt unregistered [ 4062.538884] Key type ._llcrypt registered [ 4062.539888] Key type .llcrypt registered [ 4062.570945] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 4062.882201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4062.920396] alg: No test for adler32 (adler32-zlib) [ 4063.753144] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 4063.832608] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 4065.407172] Key type lgssc registered [ 4065.766047] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4068.310074] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4068.315455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4083.743323] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4088.863221] LustreError: 179232:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.145@tcp: failed processing log, type 4: rc = -110 [ 4093.983393] LustreError: 179232:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4093.988457] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 4093.990629] LustreError: MGC192.168.206.145@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 4093.994037] LustreError: 179232:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-OST0000: -22 [ 4093.996715] LustreError: 179232:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -22 [ 4093.998792] LustreError: 179232:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 4094.017155] Lustre: server umount lustre-OST0000 complete [ 4094.018636] LustreError: 179232:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4096.942300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4098.083683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4099.485462] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4102.442562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4102.588397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4104.055382] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4106.527889] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4108.504718] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4110.903599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4110.976152] LustreError: 181542:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4110.979467] Lustre: 181542:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.206.145@tcp: invalid local config log lustre-OST0000: rc = -22 [ 4111.020308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4113.095038] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4115.536906] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4117.983895] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4117.988069] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4117.992790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4118.496216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4118.498035] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4122.327101] Lustre: server umount lustre-MDT0000 complete [ 4123.521115] LustreError: 179710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789188986 with bad export cookie 4841819108392245010 [ 4123.524202] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4123.526123] LustreError: 179710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4123.632377] Lustre: server umount lustre-MDT0001 complete [ 4135.203275] Lustre: server umount lustre-OST0000 complete [ 4138.391532] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 4141.655784] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 4152.857816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4152.930024] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4152.939303] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4152.974072] Lustre: lustre-MDT0000: new disk, initializing [ 4152.996898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4153.002856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4154.217600] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4158.217691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4158.262327] Lustre: 185061:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4158.282888] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4158.284454] Lustre: Skipped 1 previous similar message [ 4158.315871] Lustre: lustre-MDT0001: new disk, initializing [ 4158.343816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4158.354538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4158.357073] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4159.591978] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4163.138307] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4164.887565] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4167.009447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4167.105073] Lustre: lustre-OST0000: new disk, initializing [ 4167.106396] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4167.108510] Lustre: 187026:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4167.137458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4167.273905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4167.276692] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4167.314413] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4168.854189] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4172.358108] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4173.851201] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4173.927339] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4175.355202] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4175.444068] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4177.888414] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4177.890571] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4177.890647] LustreError: Skipped 1 previous similar message [ 4177.894392] Lustre: Skipped 1 previous similar message [ 4177.894890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4177.900351] Lustre: Skipped 3 previous similar messages [ 4182.225664] Lustre: server umount lustre-OST0000 complete [ 4184.031801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4184.034261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4184.038213] Lustre: Skipped 1 previous similar message [ 4184.039679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4188.128164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4188.128563] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4189.658772] Lustre: server umount lustre-MDT0000 complete [ 4190.932262] LustreError: 185854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789189053 with bad export cookie 4841819108392245710 [ 4190.935600] LustreError: 185854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4190.935633] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4191.048785] Lustre: server umount lustre-MDT0001 complete [ 4193.996293] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 00:57:36 (1789189056) [ 4200.107667] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 4201.140404] Key type lgssc unregistered [ 4201.278397] LNet: 189967:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4201.282195] LNetError: 189967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4201.290329] LNet: Removed LNI 192.168.206.145@tcp [ 4201.593118] Key type .llcrypt unregistered [ 4201.594206] Key type ._llcrypt unregistered [ 4209.187734] Key type ._llcrypt registered [ 4209.188703] Key type .llcrypt registered [ 4209.228787] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 4209.552708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4209.563177] alg: No test for adler32 (adler32-zlib) [ 4210.422564] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 4210.512537] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 4212.103198] Key type lgssc registered [ 4212.472957] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4214.927518] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4214.933513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4216.058441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4217.307168] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4219.980408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4220.096937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4221.387248] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4223.664171] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4225.448438] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4227.888777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4227.960121] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4227.962177] LustreError: 193018:0:(llog.c:1696:llog_backup()) MGC192.168.206.145@tcp: failed to backup log lustre-OST0000: rc = -28 [ 4227.965440] Lustre: 193018:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.206.145@tcp: can't backup local config lustre-OST0000: rc = -28 [ 4228.000028] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4229.813420] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4232.067332] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4235.233406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4235.234575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4235.236626] Lustre: Skipped 2 previous similar messages [ 4235.238593] Lustre: Skipped 1 previous similar message [ 4240.352147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4240.353776] Lustre: Skipped 3 previous similar messages [ 4245.472888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4245.476025] Lustre: Skipped 2 previous similar messages [ 4247.007112] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4247.068876] Lustre: server umount lustre-MDT0000 complete [ 4248.296209] LustreError: 191138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789189111 with bad export cookie 17048648425122919579 [ 4248.299840] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4248.303270] LustreError: 191138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4248.405482] Lustre: server umount lustre-MDT0001 complete [ 4259.989501] Lustre: server umount lustre-OST0000 complete [ 4262.811702] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 4265.739181] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 4276.858446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4276.952232] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4276.962421] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4276.999671] Lustre: lustre-MDT0000: new disk, initializing [ 4277.023065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4277.032682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4278.293981] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4282.319646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4282.362170] Lustre: 196538:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4282.381382] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4282.382412] Lustre: Skipped 1 previous similar message [ 4282.383677] LustreError: 197321:0:(llog.c:1696:llog_backup()) MGC192.168.206.145@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 4282.386097] LustreError: 197321:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 4282.388536] Lustre: 197321:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.145@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 4282.396669] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4282.399820] Lustre: Skipped 1 previous similar message [ 4282.438635] Lustre: lustre-MDT0001: new disk, initializing [ 4282.467651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4282.477861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4282.481119] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4283.712056] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4287.223507] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4289.015929] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4291.152455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4291.244119] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4291.245075] Lustre: Skipped 1 previous similar message [ 4291.246199] LustreError: 198482:0:(llog.c:1696:llog_backup()) MGC192.168.206.145@tcp: failed to backup log lustre-OST0000: rc = -28 [ 4291.248464] LustreError: 198482:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 4291.250071] Lustre: 198482:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.145@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 4291.252883] Lustre: 198482:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 4291.260118] Lustre: lustre-OST0000: new disk, initializing [ 4291.261436] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4291.263530] Lustre: 198506:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4291.290102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4291.521808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4291.524890] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4291.551792] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4293.040567] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4296.743099] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4298.191761] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4298.264295] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4299.674581] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4299.746464] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4301.792491] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4301.794588] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4301.796072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4301.797598] Lustre: Skipped 1 previous similar message [ 4306.513186] Lustre: server umount lustre-OST0000 complete [ 4307.935909] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4307.939381] LustreError: Skipped 1 previous similar message [ 4307.941570] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4307.946531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4307.949069] Lustre: Skipped 1 previous similar message [ 4312.032958] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4317.152168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4317.154359] Lustre: Skipped 3 previous similar messages [ 4322.271112] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4322.333783] Lustre: server umount lustre-MDT0000 complete [ 4323.559613] LustreError: 196533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789189186 with bad export cookie 17048648425122920279 [ 4323.564786] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4323.697701] Lustre: server umount lustre-MDT0001 complete [ 4326.666916] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 00:59:49 (1789189189) [ 4332.359124] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 4335.686476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4335.803756] LustreError: 201631:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4335.810305] LustreError: 201631:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4335.815792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4337.031038] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4339.663437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4341.030841] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4341.910423] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4343.930936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4345.805094] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4348.357186] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4348.408505] Lustre: lustre-OST0001: new disk, initializing [ 4348.410212] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4348.412795] Lustre: 203937:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4350.164857] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4351.040491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4351.043637] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4351.073130] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4355.424199] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4360.983761] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4361.665374] Lustre: DEBUG MARKER: ADD OST3 [ 4363.222749] loop: module loaded [ 4363.238636] loop0: detected capacity change from 0 to 204800000 [ 4363.319925] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4363.323457] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4363.326461] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4365.284260] loop0: detected capacity change from 0 to 204800000 [ 4367.095266] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4367.155677] Lustre: lustre-OST0003: new disk, initializing [ 4367.157521] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 4367.159945] Lustre: 205669:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4368.263545] Lustre: DEBUG MARKER: STOP OST3 [ 4368.339380] Lustre: *** cfs_fail_loc=2109, val=2*** [ 4368.381715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 4368.384625] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 4368.394823] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 4368.937681] Lustre: Failing over lustre-OST0003 [ 4368.992323] Lustre: server umount lustre-OST0003 complete [ 4369.246048] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4370.400080] LustreError: 205908:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 4373.222910] Lustre: DEBUG MARKER: START OST3 again [ 4373.475751] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4373.477899] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4373.479822] LustreError: 203115:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4373.481554] Lustre: Skipped 1 previous similar message [ 4375.327728] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4375.400489] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 4375.402989] Lustre: Skipped 4 previous similar messages [ 4375.406912] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 4376.865905] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4377.045775] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4377.046135] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4377.119618] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4390.383748] Lustre: Failing over lustre-OST0003 [ 4390.422300] Lustre: server umount lustre-OST0003 complete [ 4390.883347] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4390.885772] LustreError: 203115:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4390.887743] Lustre: Skipped 1 previous similar message [ 4390.892387] LustreError: 203115:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4396.003700] LustreError: 203114:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4396.008867] LustreError: 203114:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4404.195915] LustreError: 203113:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4404.196233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4404.200630] LustreError: 203113:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4404.202313] Lustre: Skipped 5 previous similar messages [ 4408.992684] Lustre: server umount lustre-MDT0000 complete [ 4410.186309] LustreError: 204755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789189273 with bad export cookie 17048648425122921147 [ 4410.189605] LustreError: 204755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4410.190668] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4414.431905] LustreError: 205680:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4414.432667] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4414.436588] LustreError: 205680:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 4414.439770] Lustre: Skipped 4 previous similar messages [ 4416.426291] Lustre: server umount lustre-MDT0001 complete [ 4417.739302] Lustre: server umount lustre-OST0000 complete [ 4418.978684] Lustre: server umount lustre-OST0001 complete [ 4421.110248] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 4423.725601] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 4435.798890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4435.870648] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4435.879236] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4435.911488] Lustre: lustre-MDT0000: new disk, initializing [ 4435.934197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4437.124329] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4441.286797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4441.330218] Lustre: 211090:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4441.420858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4441.423835] Lustre: Skipped 1 previous similar message [ 4441.435206] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4442.804204] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4446.583480] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4448.493075] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4450.746592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4450.850907] Lustre: 213057:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4452.532056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4452.535920] Lustre: Skipped 1 previous similar message [ 4452.538047] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4452.565896] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4452.698781] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4456.185136] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4457.549946] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4457.616219] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4458.948158] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4459.023666] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4461.707483] Lustre: server umount lustre-OST0000 complete [ 4463.072670] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4463.076047] Lustre: Skipped 1 previous similar message [ 4463.077930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4463.079959] Lustre: Skipped 5 previous similar messages [ 4477.407154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4477.477747] Lustre: server umount lustre-MDT0000 complete [ 4478.239119] Lustre: 190472:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789189325/real 1789189325] req@ffff94d5d67e2300 x1876100721595008/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789189341 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4478.251364] LustreError: 211101:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4478.257376] LustreError: 211101:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4478.758479] LustreError: 211082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789189341 with bad export cookie 17048648425122923401 [ 4478.761683] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4478.763776] LustreError: 211082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4478.887136] Lustre: server umount lustre-MDT0001 complete [ 4480.132113] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 01:02:22 (1789189342) [ 4485.387818] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 4488.791803] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 4500.721094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4500.803806] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4500.814636] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4500.816516] Lustre: Skipped 3 previous similar messages [ 4500.850549] Lustre: lustre-MDT0000: new disk, initializing [ 4500.851855] Lustre: Skipped 2 previous similar messages [ 4500.876881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4502.272222] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4506.538721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4506.583724] Lustre: 217656:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4506.689176] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4508.076412] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4511.846635] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4513.791642] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4516.131227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4516.235651] Lustre: 219622:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4517.667197] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4518.295152] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4522.090700] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4523.445423] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4523.509389] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4524.709946] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4524.773126] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4528.098026] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4528.098492] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4528.099069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4528.099074] Lustre: Skipped 4 previous similar messages [ 4528.100351] LustreError: Skipped 1 previous similar message [ 4528.103386] Lustre: Skipped 3 previous similar messages [ 4531.408541] Lustre: server umount lustre-OST0000 complete [ 4547.039153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4547.104279] Lustre: server umount lustre-MDT0000 complete [ 4548.326321] LustreError: 217649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789189411 with bad export cookie 17048648425122924283 [ 4548.329285] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4548.329817] LustreError: 217649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4548.446171] Lustre: server umount lustre-MDT0001 complete [ 4551.217469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4551.358080] LustreError: 221836:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4552.716655] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4555.542022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4557.020043] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4559.450665] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4561.336187] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4563.601662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4565.670401] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4568.139974] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4586.464602] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4591.122990] Lustre: server umount lustre-OST0000 complete [ 4598.552923] Lustre: server umount lustre-MDT0000 complete [ 4599.856149] LustreError: 223677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789189462 with bad export cookie 17048648425122925165 [ 4599.859220] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4599.859612] LustreError: 223677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4599.982646] Lustre: server umount lustre-MDT0001 complete [ 4701.213073] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 4702.250167] Key type lgssc unregistered [ 4702.379302] LNet: 225521:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4702.381510] LNetError: 225521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4702.391273] LNet: Removed LNI 192.168.206.145@tcp [ 4702.720128] Key type .llcrypt unregistered [ 4702.722209] Key type ._llcrypt unregistered [ 4710.912203] Key type ._llcrypt registered [ 4710.913936] Key type .llcrypt registered [ 4710.953266] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 4717.175814] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 4717.545718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4717.576421] alg: No test for adler32 (adler32-zlib) [ 4718.445760] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 4718.539738] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 4720.127147] Key type lgssc registered [ 4720.474936] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4731.650838] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4731.656590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4732.738109] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4732.747529] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4732.782878] Lustre: lustre-MDT0000: new disk, initializing [ 4732.804199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4732.810905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4734.156748] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4738.280766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4738.332206] Lustre: 228840:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4738.363672] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4738.365982] Lustre: Skipped 1 previous similar message [ 4738.411419] Lustre: lustre-MDT0001: new disk, initializing [ 4738.451758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4738.465412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4738.470603] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4739.854163] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4743.600596] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4745.429629] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4747.734191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4747.834023] Lustre: lustre-OST0000: new disk, initializing [ 4747.835530] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4747.837907] Lustre: 230808:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4747.866118] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4749.893227] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4753.400140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4753.403447] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4753.435038] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4753.610821] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4755.017047] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4755.090934] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4756.441910] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4756.512686] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4758.497042] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4758.498860] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4758.499714] LustreError: Skipped 1 previous similar message [ 4758.500168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4758.505567] Lustre: Skipped 1 previous similar message [ 4759.187766] Lustre: server umount lustre-OST0000 complete [ 4763.616780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4763.617581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4763.620232] Lustre: Skipped 1 previous similar message [ 4763.623861] Lustre: Skipped 2 previous similar messages [ 4766.610408] Lustre: server umount lustre-MDT0000 complete [ 4767.842076] LustreError: 228834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789189630 with bad export cookie 6187176826421729057 [ 4767.844767] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4767.847884] LustreError: 228834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4767.965075] Lustre: server umount lustre-MDT0001 complete [ 4774.192974] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 4777.539101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4777.686058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4779.053854] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4781.946945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4782.060574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4783.497606] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4784.465258] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4786.675260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4788.684274] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4791.430467] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4791.496403] Lustre: lustre-OST0001: new disk, initializing [ 4791.498313] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4791.500733] Lustre: 236042:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4791.532960] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4791.535126] Lustre: Skipped 1 previous similar message [ 4793.475426] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4798.893613] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4799.499039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4799.502279] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4799.540466] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4802.605160] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4806.447848] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 01:07:49 (1789189669) [ 4807.648499] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4807.653225] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4807.662064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4809.696488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4809.699225] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4809.704552] Lustre: Skipped 2 previous similar messages [ 4813.209269] Lustre: server umount lustre-MDT0000 complete [ 4814.424236] LustreError: 233718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789189677 with bad export cookie 6187176826421729925 [ 4814.427252] LustreError: 233718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4814.428482] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4814.538386] Lustre: server umount lustre-MDT0001 complete [ 4825.986498] Lustre: server umount lustre-OST0000 complete [ 4837.540297] Lustre: server umount lustre-OST0001 complete [ 4839.684774] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 4842.987431] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 4854.410727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4854.488804] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4854.497403] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4854.547134] Lustre: lustre-MDT0000: new disk, initializing [ 4854.569356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4854.580590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4856.013376] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4860.308364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4860.359196] Lustre: 239978:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4860.471742] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4861.941718] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4865.664194] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4867.577214] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4869.777330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4869.874788] Lustre: lustre-OST0000: new disk, initializing [ 4869.876126] Lustre: Skipped 1 previous similar message [ 4869.877454] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4869.879127] Lustre: Skipped 2 previous similar messages [ 4869.880914] Lustre: 241950:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4871.675051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4871.678302] Lustre: Skipped 1 previous similar message [ 4871.679777] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4871.720036] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4871.803517] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4875.528308] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4876.990369] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4877.080980] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4878.577152] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4878.657632] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4881.357450] Lustre: server umount lustre-OST0000 complete [ 4885.984055] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4885.987620] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4885.993236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4885.995961] Lustre: Skipped 3 previous similar messages [ 4893.182316] Lustre: server umount lustre-MDT0000 complete [ 4894.453307] LustreError: 239971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789189757 with bad export cookie 6187176826421732046 [ 4894.456752] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4894.457194] LustreError: 239971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4904.983528] Lustre: server umount lustre-MDT0001 complete [ 4907.800472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4907.945359] LustreError: 244164:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4907.951878] LustreError: 244164:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4907.969260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4907.972893] Lustre: Skipped 2 previous similar messages [ 4909.357910] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4912.211131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4913.703218] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4916.203217] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4918.106651] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4920.454771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4922.547122] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4924.995896] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5071.329642] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5071.331624] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5071.334545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5071.340241] Lustre: Skipped 2 previous similar messages [ 5071.342200] Lustre: Skipped 4 previous similar messages [ 5072.781635] Lustre: server umount lustre-OST0000 complete [ 5076.449887] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5080.212668] Lustre: server umount lustre-MDT0000 complete [ 5081.490815] LustreError: 244189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789189944 with bad export cookie 6187176826421732928 [ 5081.494210] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5081.498155] LustreError: 244189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5081.629669] Lustre: server umount lustre-MDT0001 complete [ 5187.998665] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 5189.131195] Key type lgssc unregistered [ 5189.277498] LNet: 247770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5189.281182] LNetError: 247770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5189.288407] LNet: Removed LNI 192.168.206.145@tcp [ 5189.644135] Key type .llcrypt unregistered [ 5189.645296] Key type ._llcrypt unregistered [ 5198.878265] Key type ._llcrypt registered [ 5198.879421] Key type .llcrypt registered [ 5198.919965] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 5205.961817] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 5206.331885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5206.372061] alg: No test for adler32 (adler32-zlib) [ 5207.232460] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 5207.333386] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 5208.927185] Key type lgssc registered [ 5209.265981] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5224.831826] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 5228.673898] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5228.684992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5229.810126] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5229.822180] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5229.872982] Lustre: lustre-MDT0000: new disk, initializing [ 5229.902335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5229.911100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5231.659654] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 5236.821324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5236.869060] Lustre: 251807:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5236.896288] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5236.898221] Lustre: Skipped 1 previous similar message [ 5236.936542] Lustre: lustre-MDT0001: new disk, initializing [ 5236.971427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5236.983771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5236.987542] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5238.701182] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 5241.344847] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5243.753887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5243.875717] Lustre: lustre-OST0000: new disk, initializing [ 5243.877585] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5243.880367] Lustre: 253440:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5243.914928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5245.634456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5245.638666] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5245.691584] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5245.987675] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 5250.253064] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5250.331734] Lustre: lustre-OST0001: new disk, initializing [ 5250.333438] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5250.336213] Lustre: 254313:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5250.371637] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5251.803468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5251.809579] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5251.848934] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5252.818451] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 5258.754099] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5264.011147] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5267.426053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5267.430089] Lustre: Skipped 1 previous similar message [ 5267.432009] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5267.433988] Lustre: Skipped 1 previous similar message [ 5272.544923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5272.548225] Lustre: Skipped 1 previous similar message [ 5273.413869] Lustre: server umount lustre-MDT0000 complete [ 5274.909405] LustreError: 251799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789190137 with bad export cookie 6258003551707149139 [ 5274.913587] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5274.914760] LustreError: 251799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5275.031604] Lustre: server umount lustre-MDT0001 complete [ 5286.925598] Lustre: server umount lustre-OST0000 complete [ 5298.638796] Lustre: server umount lustre-OST0001 complete [ 5304.959601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5309.475071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5312.706864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5316.243724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5322.689986] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 5326.503976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5326.510896] Lustre: lustre-MDT0000: reset Object Index mappings [ 5326.658708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5328.026261] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 5331.004031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5331.013756] Lustre: lustre-MDT0001: reset Object Index mappings [ 5332.519693] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 5333.574940] Lustre: 260455:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5335.978670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5336.121603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5336.123497] Lustre: Skipped 1 previous similar message [ 5337.126689] LustreError: 260809:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5337.130946] LustreError: 260809:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5337.131982] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5338.065689] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 5340.811045] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5341.938085] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 5342.829945] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 5345.934359] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5347.059728] Lustre: 262324:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5351.904069] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5351.906653] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5351.911083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5351.912819] Lustre: Skipped 1 previous similar message [ 5354.199564] Lustre: server umount lustre-MDT0000 complete [ 5355.430147] LustreError: 259293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789190218 with bad export cookie 6258003551707155915 [ 5355.433213] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5355.434561] LustreError: 259293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5355.575509] Lustre: server umount lustre-MDT0001 complete [ 5366.724325] Lustre: server umount lustre-OST0000 complete [ 5378.141199] Lustre: server umount lustre-OST0001 complete [ 5386.016432] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 01:17:28 (1789190248) [ 5391.973748] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 5396.027426] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 5409.554787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5409.647875] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5409.659182] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5409.695936] Lustre: lustre-MDT0000: new disk, initializing [ 5409.718576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5409.721485] Lustre: Skipped 1 previous similar message [ 5409.727213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5411.147626] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 5415.573347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5415.617254] Lustre: 266572:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5415.718728] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5417.154434] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 5421.492742] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5423.825804] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5426.724765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5426.872054] Lustre: lustre-OST0000: new disk, initializing [ 5426.874173] Lustre: Skipped 1 previous similar message [ 5426.876531] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5426.879159] Lustre: Skipped 2 previous similar messages [ 5426.882039] Lustre: 268537:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5428.631442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5428.636072] Lustre: Skipped 1 previous similar message [ 5428.639175] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5428.700522] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5429.772918] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 5434.466354] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5435.958624] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5436.053747] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5437.754778] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5437.860989] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5438.946096] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5438.947173] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5438.948674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5438.948680] Lustre: Skipped 4 previous similar messages [ 5438.950218] Lustre: Skipped 4 previous similar messages [ 5438.952776] LustreError: Skipped 1 previous similar message [ 5444.066048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5444.068588] Lustre: Skipped 2 previous similar messages [ 5444.780910] Lustre: server umount lustre-OST0000 complete [ 5446.624107] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5446.628040] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5452.776992] Lustre: server umount lustre-MDT0000 complete [ 5454.100607] LustreError: 266562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789190316 with bad export cookie 6258003551707157987 [ 5454.104679] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5454.108414] LustreError: 266562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5454.251957] Lustre: server umount lustre-MDT0001 complete [ 5461.524831] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 5465.374656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5465.553375] LustreError: 271466:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5465.565516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5465.567635] Lustre: Skipped 2 previous similar messages [ 5466.927134] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 5469.814911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5471.309595] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 5472.264002] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5474.606369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5476.868479] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 5480.244783] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5480.332445] Lustre: lustre-OST0001: new disk, initializing [ 5480.334783] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5480.337997] Lustre: 273772:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5482.065211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5482.068359] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5482.106041] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5482.486571] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 5487.898641] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5489.408704] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5491.296153] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 5491.300693] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 5495.776179] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5495.780742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5495.787926] Lustre: Skipped 1 previous similar message [ 5495.791305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5495.795141] Lustre: Skipped 3 previous similar messages [ 5500.919524] Lustre: server umount lustre-MDT0000 complete [ 5502.402358] LustreError: 271446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789190365 with bad export cookie 6258003551707158869 [ 5502.406062] LustreError: 271446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5502.406194] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5502.547587] Lustre: server umount lustre-MDT0001 complete [ 5514.502275] Lustre: server umount lustre-OST0000 complete [ 5526.061266] Lustre: server umount lustre-OST0001 complete [ 5533.036153] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 5537.427098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5537.583640] LustreError: 277113:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5537.600622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5537.602287] Lustre: Skipped 3 previous similar messages [ 5539.063764] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 5542.267963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5543.819915] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 5544.853226] Lustre: 278255:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5546.990885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5548.131926] LustreError: 278607:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5548.136689] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 5548.944317] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 5551.771673] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5552.879152] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 5553.813687] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 5556.843297] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5557.980909] Lustre: 280131:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5559.601563] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 5559.603650] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 5563.943516] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 01:20:26 (1789190426) [ 5578.208547] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5578.211556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5578.216205] Lustre: Skipped 3 previous similar messages [ 5578.218463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5578.220972] Lustre: Skipped 4 previous similar messages [ 5581.995718] Lustre: server umount lustre-MDT0000 complete [ 5583.400479] LustreError: 277096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789190446 with bad export cookie 6258003551707165225 [ 5583.403427] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5583.404093] LustreError: 277096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5583.840960] LustreError: 280969:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5583.851660] LustreError: 280969:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5589.607316] Lustre: server umount lustre-MDT0001 complete [ 5597.438147] Lustre: server umount lustre-OST0000 complete [ 5605.332457] Lustre: server umount lustre-OST0001 complete [ 5607.389640] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 5611.389772] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 5627.252774] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 5630.752749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5630.848602] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5630.859875] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5630.899827] Lustre: lustre-MDT0000: new disk, initializing [ 5630.927182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5632.229451] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 5636.715837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5636.764156] Lustre: 284649:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5636.872043] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5638.298723] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 5643.791246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5643.913576] Lustre: 286282:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5645.250748] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5646.515904] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 5651.808360] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5653.930095] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5654.196978] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 5659.236120] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5665.039506] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 01:22:07 (1789190527) [ 5667.808387] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5667.812513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5667.818411] Lustre: Skipped 5 previous similar messages [ 5667.821651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5667.824838] Lustre: Skipped 7 previous similar messages [ 5671.874524] Lustre: server umount lustre-MDT0000 complete [ 5673.497338] LustreError: 284643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789190536 with bad export cookie 6258003551707173156 [ 5673.505592] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5673.677068] Lustre: server umount lustre-MDT0001 complete [ 5684.376205] Lustre: server umount lustre-OST0000 complete [ 5696.033497] Lustre: server umount lustre-OST0001 complete [ 5698.971202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5699.137769] LustreError: 289396:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5699.142908] LustreError: 289396:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5699.171989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5699.174017] Lustre: Skipped 7 previous similar messages [ 5700.583723] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 5704.328069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5706.172185] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 5709.138606] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5711.205420] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5713.922256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5716.201246] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 5718.607128] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5720.549119] LustreError: 291222:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5720.554717] LustreError: 291222:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5735.821428] loop1: detected capacity change from 0 to 204800000 [ 5735.925709] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5735.930365] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5735.936521] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5738.172521] loop1: detected capacity change from 0 to 204800000 [ 5740.471921] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5740.512222] LustreError: 289386:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs2145: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 5740.517934] LustreError: 289386:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs2145 [ 5740.520522] LustreError: 289386:0:(mgs_handler.c:517:mgs_target_reg()) fs2145-MDT0001: Error checking target index: rc = -13 [ 5740.524931] LustreError: fs2145-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 5740.528669] LustreError: 293131:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [ 5740.531742] LustreError: 293131:0:(tgt_mount.c:2128:server_put_super()) no obd fs2145-MDT0001 [ 5740.534406] LustreError: 293131:0:(tgt_mount.c:116:server_deregister_mount()) fs2145-MDT0001 not registered [ 5740.548326] Lustre: server umount fs2145-MDT0001 complete [ 5740.550025] LustreError: 293131:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 5743.620823] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5743.654604] Lustre: Setting parameter fs2145-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs2145-MDT0001 [ 5743.657408] Lustre: Skipped 2 previous similar messages [ 5743.674277] Lustre: srv-fs2145-MDT0001: No data found on store. Initialize space. [ 5743.676967] Lustre: Skipped 4 previous similar messages [ 5743.718953] Lustre: fs2145-MDT0001: new disk, initializing [ 5743.720282] Lustre: Skipped 3 previous similar messages [ 5743.751771] Lustre: srv-fs2145-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 5743.753991] Lustre: 293597:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs2145-MDT0001-mdtlov: can't create new llog: rc = -115 [ 5743.756457] LustreError: 293597:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs2145-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 5745.108352] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 5748.407354] Lustre: server umount fs2145-MDT0001 complete [ 5753.321385] LustreError: 291223:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5753.332098] LustreError: 291223:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 18 previous similar messages [ 5754.024893] loop1: detected capacity change from 0 to 204800000 [ 5756.706830] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5756.894414] Lustre: 295166:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs2145-MDT0001-mdtlov: can't create new llog: rc = -115 [ 5756.897125] LustreError: 295166:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs2145-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 5758.802132] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 5761.175471] Lustre: server umount fs2145-MDT0001 complete [ 5765.209741] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.45@tcp (stopping) [ 5765.213983] Lustre: Skipped 4 previous similar messages [ 5766.112120] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5766.115853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5766.123278] Lustre: Skipped 3 previous similar messages [ 5771.112858] Lustre: server umount lustre-MDT0000 complete [ 5775.001804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5775.039331] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5776.838339] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 5779.965636] loop1: detected capacity change from 0 to 204800000 [ 5780.461965] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5782.342252] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5782.505269] Lustre: 297608:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs2145-MDT0001-mdtlov: can't create new llog: rc = -115 [ 5782.509894] LustreError: 297608:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs2145-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 5783.934814] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 5786.383900] Lustre: server umount fs2145-MDT0001 complete [ 5800.543566] Lustre: server umount lustre-OST0000 complete [ 5806.050442] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5806.055969] LustreError: Skipped 1 previous similar message [ 5816.287159] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5816.384170] Lustre: server umount lustre-MDT0000 complete [ 5818.084724] LustreError: 289378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789190680 with bad export cookie 6258003551707175830 [ 5818.088790] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5818.091923] LustreError: 289378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 5818.291653] Lustre: server umount lustre-MDT0001 complete [ 5822.179315] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 5823.401494] Key type lgssc unregistered [ 5823.551538] LNet: 299947:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5823.556622] LNetError: 299947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5823.565609] LNet: Removed LNI 192.168.206.145@tcp [ 5824.034115] Key type .llcrypt unregistered [ 5824.035281] Key type ._llcrypt unregistered [ 5832.209637] Key type ._llcrypt registered [ 5832.210735] Key type .llcrypt registered [ 5832.249768] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 5832.577770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5832.596096] alg: No test for adler32 (adler32-zlib) [ 5833.447855] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 5833.538119] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 5835.119212] Key type lgssc registered [ 5835.547991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5838.845927] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5838.854528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5840.040753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5841.754604] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 5845.611744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5845.743956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5847.179486] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 5849.955058] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5851.922158] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5854.524287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5854.663227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5855.718931] LustreError: 303016:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5855.726163] LustreError: 303016:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5856.844672] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 5859.451284] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5860.839225] LustreError: 303199:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5864.937421] LustreError: 303196:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5864.949193] LustreError: 303196:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5870.054631] LustreError: 303016:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5870.060376] LustreError: 303016:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5872.121225] loop1: detected capacity change from 0 to 204800000 [ 5872.291606] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5872.297498] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5872.302930] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5874.685774] loop1: detected capacity change from 0 to 204800000 [ 5875.176185] LustreError: 303018:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5875.183435] LustreError: 303018:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5877.143692] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5877.230618] Lustre: fs32746-OST0000: new disk, initializing [ 5877.233481] Lustre: srv-fs32746-OST0000: No data found on store. Initialize space. [ 5877.236652] Lustre: 304924:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5877.264539] Lustre: fs32746-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5879.822093] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 5883.949781] Lustre: server umount fs32746-OST0000 complete [ 5885.411210] LustreError: 303018:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5885.418764] LustreError: 303018:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 5889.427555] loop1: detected capacity change from 0 to 204800000 [ 5891.751539] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5891.851759] Lustre: fs32746-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5893.992820] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 5897.367534] loop2: detected capacity change from 0 to 204800000 [ 5897.449673] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5897.453335] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5897.457770] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5899.502284] loop2: detected capacity change from 0 to 204800000 [ 5901.742060] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5901.791627] LustreError: fs32746-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 5901.795697] LustreError: fs32746-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 5901.799791] LustreError: 307963:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [ 5901.803028] LustreError: 307963:0:(tgt_mount.c:2128:server_put_super()) no obd fs32746-OST0001 [ 5901.805908] LustreError: 307963:0:(tgt_mount.c:116:server_deregister_mount()) fs32746-OST0001 not registered [ 5901.823055] Lustre: server umount fs32746-OST0001 complete [ 5901.824736] LustreError: 307963:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 5905.896826] LustreError: 303016:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5905.901243] LustreError: 303016:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 5906.235746] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5906.297126] Lustre: fs32746-OST0001: new disk, initializing [ 5906.299505] Lustre: srv-fs32746-OST0001: No data found on store. Initialize space. [ 5906.302830] Lustre: 308514:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5906.332845] Lustre: fs32746-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5908.537294] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 5912.514448] Lustre: server umount fs32746-OST0000 complete [ 5916.814316] Lustre: server umount fs32746-OST0001 complete [ 5921.681200] loop1: detected capacity change from 0 to 204800000 [ 5923.854659] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5923.946929] Lustre: fs32746-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5925.997392] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 5928.512657] loop2: detected capacity change from 0 to 204800000 [ 5930.728789] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5933.100364] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 5936.769663] Lustre: server umount fs32746-OST0000 complete [ 5940.941572] Lustre: server umount fs32746-OST0001 complete [ 5941.735883] LustreError: 303199:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5941.746223] LustreError: 303199:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 20 previous similar messages [ 5946.858879] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 5946.860174] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5946.864217] LustreError: Skipped 1 previous similar message [ 5946.877942] Lustre: Skipped 1 previous similar message [ 5946.881884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5946.889222] Lustre: Skipped 1 previous similar message [ 5951.782591] Lustre: server umount lustre-OST0000 complete [ 5957.089805] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5957.090985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5957.093309] Lustre: Skipped 1 previous similar message [ 5957.098361] Lustre: Skipped 1 previous similar message [ 5959.793482] Lustre: server umount lustre-MDT0000 complete [ 5961.456989] LustreError: 301173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789190824 with bad export cookie 15883087280613788541 [ 5961.464519] LustreError: 301173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5961.466018] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5961.643922] Lustre: server umount lustre-MDT0001 complete [ 5966.121806] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 5967.255576] Key type lgssc unregistered [ 5967.389518] LNet: 314423:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5967.393391] LNetError: 314423:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5967.406537] LNet: Removed LNI 192.168.206.145@tcp [ 5967.817126] Key type .llcrypt unregistered [ 5967.818397] Key type ._llcrypt unregistered [ 5980.688147] Key type ._llcrypt registered [ 5980.689254] Key type .llcrypt registered [ 5980.732463] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 5988.374857] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 5988.755689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5988.771990] alg: No test for adler32 (adler32-zlib) [ 5989.654171] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 5989.756560] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 5991.343187] Key type lgssc registered [ 5991.761194] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6008.171600] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6008.177267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6009.276887] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6009.291875] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6009.342934] Lustre: lustre-MDT0000: new disk, initializing [ 6009.412983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6009.424276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6011.049337] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6015.710641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6015.770464] Lustre: 318248:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6015.811463] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6015.815809] Lustre: Skipped 1 previous similar message [ 6015.866887] Lustre: lustre-MDT0001: new disk, initializing [ 6015.919740] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6015.940917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6015.946428] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6017.742418] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6022.218081] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6024.637451] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6027.740042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6027.889937] Lustre: lustre-OST0000: new disk, initializing [ 6027.892265] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6027.895485] Lustre: 320218:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6027.940994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6030.667989] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6035.161804] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6035.989873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6035.995708] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6036.057742] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6036.105533] Lustre: Mounted lustre-client - version 2.17.58_40_g357d090 [ 6036.527688] Lustre: Unmounted lustre-client [ 6041.057084] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6041.057575] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6041.063944] LustreError: Skipped 1 previous similar message [ 6041.073632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6041.079504] Lustre: Skipped 1 previous similar message [ 6045.988477] Lustre: server umount lustre-OST0000 complete [ 6051.809671] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6051.809794] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6051.811189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6051.817907] Lustre: Skipped 2 previous similar messages [ 6053.689421] Lustre: server umount lustre-MDT0000 complete [ 6055.323991] LustreError: 318243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789190918 with bad export cookie 12655058350987657245 [ 6055.330243] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6055.330423] LustreError: 318243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6055.470324] Lustre: server umount lustre-MDT0001 complete [ 6059.947090] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 6061.333805] Key type lgssc unregistered [ 6095.263259] LNet: 1 local NIs in recovery (showing 1): 192.168.206.145@tcp [ 6095.265654] LNet: 322379:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6095.268927] LNetError: 322379:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6165.925331] LNet: Removed LNI 192.168.206.145@tcp [ 6166.202877] Key type .llcrypt unregistered [ 6166.204166] Key type ._llcrypt unregistered [ 6173.490264] Key type ._llcrypt registered [ 6173.491238] Key type .llcrypt registered [ 6173.527444] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 6181.321931] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 6181.638024] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6181.648039] alg: No test for adler32 (adler32-zlib) [ 6182.511731] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 6182.601563] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 6184.183177] Key type lgssc registered [ 6184.551092] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6196.326426] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 01:30:58 (1789191058) [ 6198.870745] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6198.878754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6199.985548] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6199.995561] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6200.032892] Lustre: lustre-MDT0000: new disk, initializing [ 6200.055089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6200.061884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6201.769926] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6206.328907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6206.384560] Lustre: 325573:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6206.416604] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6206.419145] Lustre: Skipped 1 previous similar message [ 6206.461842] Lustre: lustre-MDT0001: new disk, initializing [ 6206.499877] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6206.515214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6206.518826] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6207.926685] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6212.577213] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6214.756285] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6217.331140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6217.477808] Lustre: lustre-OST0000: new disk, initializing [ 6217.479859] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6217.482724] Lustre: 327544:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6217.520190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6219.739278] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6224.009973] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6226.446965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6226.456350] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6226.508425] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6236.648631] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6236.649050] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6236.652536] LustreError: Skipped 1 previous similar message [ 6236.658891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6236.663828] Lustre: Skipped 1 previous similar message [ 6241.646861] Lustre: server umount lustre-OST0000 complete [ 6246.881298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6246.882646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6246.884424] Lustre: Skipped 2 previous similar messages [ 6246.888677] Lustre: Skipped 1 previous similar message [ 6249.590735] Lustre: server umount lustre-MDT0000 complete [ 6251.169277] LustreError: 325568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789191114 with bad export cookie 8919006230130854481 [ 6251.171636] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6251.174620] LustreError: 325568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6251.298225] Lustre: server umount lustre-MDT0001 complete [ 6254.964569] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 6255.930763] Key type lgssc unregistered [ 6256.054362] LNet: 329449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6256.056455] LNetError: 329449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6256.065299] LNet: Removed LNI 192.168.206.145@tcp [ 6256.368360] Key type .llcrypt unregistered [ 6256.369450] Key type ._llcrypt unregistered [ 6264.941636] Key type ._llcrypt registered [ 6264.942744] Key type .llcrypt registered [ 6264.986217] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 6265.364634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6265.392494] alg: No test for adler32 (adler32-zlib) [ 6266.272605] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 6266.373928] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 6267.967245] Key type lgssc registered [ 6268.581664] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6272.123192] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6272.137659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6273.314521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6275.063386] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6278.704311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6278.900447] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6280.735588] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6284.041680] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6286.762429] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6290.012846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6290.197678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6292.741286] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6295.732445] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6310.332495] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6311.132439] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6313.956650] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6313.958482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6313.960276] Lustre: Skipped 1 previous similar message [ 6317.665298] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.45@tcp (stopping) [ 6317.670043] Lustre: Skipped 2 previous similar messages [ 6319.075000] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6319.077832] Lustre: Skipped 2 previous similar messages [ 6319.289618] Lustre: server umount lustre-MDT0000 complete [ 6320.983870] LustreError: 330677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789191183 with bad export cookie 10328537948787752696 [ 6320.988662] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6320.993590] LustreError: 330677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6322.767239] LustreError: 330691:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6322.777363] LustreError: 330691:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6322.778676] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.45@tcp (stopping) [ 6324.192727] LustreError: 330690:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6324.193124] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6324.209702] Lustre: Skipped 1 previous similar message [ 6325.216372] LustreError: 330691:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6327.184840] Lustre: server umount lustre-MDT0001 complete [ 6335.157831] Lustre: server umount lustre-OST0000 complete [ 6339.509760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6341.512162] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6342.906607] Lustre: 334892:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.145@tcp [ 6344.002780] Lustre: server umount MGS complete [ 6348.216823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6348.468800] LustreError: 335464:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6348.510327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6350.609092] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6353.889401] LustreError: 335465:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6354.704655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6354.857292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6356.694127] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6360.656363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6363.873338] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6367.565441] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6368.343941] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 6370.844941] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6374.716750] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 6378.467379] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6378.471800] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6378.474400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6378.476368] Lustre: Skipped 1 previous similar message [ 6378.480359] Lustre: Skipped 2 previous similar messages [ 6382.364784] Lustre: server umount lustre-OST0000 complete [ 6385.633085] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6385.641236] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6388.706078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6388.712653] Lustre: Skipped 2 previous similar messages [ 6390.318468] Lustre: server umount lustre-MDT0000 complete [ 6391.741328] LustreError: 336199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789191254 with bad export cookie 10328537948787753515 [ 6391.744015] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6391.748213] LustreError: 336199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6391.864756] Lustre: server umount lustre-MDT0001 complete [ 6396.590785] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 6397.940505] Key type lgssc unregistered [ 6398.085585] LNet: 338895:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6398.090332] LNetError: 338895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6398.103538] LNet: Removed LNI 192.168.206.145@tcp [ 6398.440489] Key type .llcrypt unregistered [ 6398.441926] Key type ._llcrypt unregistered [ 6406.345253] Key type ._llcrypt registered [ 6406.346581] Key type .llcrypt registered [ 6406.413322] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 6413.760407] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 6414.307662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6414.319931] alg: No test for adler32 (adler32-zlib) [ 6415.235768] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 6415.340616] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 6416.927303] Key type lgssc registered [ 6417.465792] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6433.600962] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6433.611406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6434.737065] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6434.753090] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6434.795563] Lustre: lustre-MDT0000: new disk, initializing [ 6434.827720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6434.834435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6436.542423] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6441.216881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6441.275729] Lustre: 342024:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6441.310990] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6441.313549] Lustre: Skipped 1 previous similar message [ 6441.355703] Lustre: lustre-MDT0001: new disk, initializing [ 6441.404456] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6441.424725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6441.429849] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6443.213725] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6447.764734] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6450.385873] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6453.790548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6453.969548] Lustre: lustre-OST0000: new disk, initializing [ 6453.971995] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6453.975368] Lustre: 343992:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6454.029975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6457.179144] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6461.849162] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6462.986257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6462.992937] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6463.047906] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6463.475754] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6463.595999] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6465.366800] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6465.474511] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6468.066510] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6468.071471] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6468.076591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6468.079122] Lustre: Skipped 1 previous similar message [ 6472.475031] Lustre: server umount lustre-OST0000 complete [ 6477.280514] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6477.285061] LustreError: Skipped 1 previous similar message [ 6477.287677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6477.295825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6477.299657] Lustre: Skipped 1 previous similar message [ 6478.305830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6478.306866] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6480.054498] Lustre: server umount lustre-MDT0000 complete [ 6481.863541] LustreError: 342016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789191344 with bad export cookie 14333487592429210366 [ 6481.870870] LustreError: 342016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6481.872699] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6482.078201] Lustre: server umount lustre-MDT0001 complete [ 6486.362110] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 01:35:48 (1789191348) [ 6494.074023] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 6498.638898] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 6515.529648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6515.682795] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6515.702642] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6515.757330] Lustre: lustre-MDT0000: new disk, initializing [ 6515.792390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6515.804101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6517.727676] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6523.516578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6523.589334] Lustre: 348745:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6523.637870] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6523.641490] Lustre: Skipped 1 previous similar message [ 6523.695664] Lustre: lustre-MDT0001: new disk, initializing [ 6523.746819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6523.767827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6523.775154] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6525.747732] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6530.588816] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6533.078822] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6535.915029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6536.052954] Lustre: lustre-OST0000: new disk, initializing [ 6536.056188] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6536.061156] Lustre: 350716:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6536.109336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6538.907879] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6541.841599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6541.847869] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6541.904626] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6543.814714] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6557.159838] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6557.160571] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6557.166347] LustreError: Skipped 1 previous similar message [ 6557.173579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6557.177677] Lustre: Skipped 2 previous similar messages [ 6561.762421] Lustre: server umount lustre-OST0000 complete [ 6564.832637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6564.837341] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6564.844704] Lustre: Skipped 1 previous similar message [ 6564.848295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6569.736903] Lustre: server umount lustre-MDT0000 complete [ 6571.400682] LustreError: 349543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789191434 with bad export cookie 14333487592429211234 [ 6571.407841] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6571.542753] Lustre: server umount lustre-MDT0001 complete [ 6576.376917] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 6577.898632] Key type lgssc unregistered [ 6578.093585] LNet: 352864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6578.097737] LNetError: 352864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6578.105622] LNet: Removed LNI 192.168.206.145@tcp [ 6578.578621] Key type .llcrypt unregistered [ 6578.580793] Key type ._llcrypt unregistered [ 6622.879706] Key type ._llcrypt registered [ 6622.880743] Key type .llcrypt registered [ 6622.917667] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 6623.023245] Key type .llcrypt unregistered [ 6623.024438] Key type ._llcrypt unregistered [ 6657.694382] Key type ._llcrypt registered [ 6657.695759] Key type .llcrypt registered [ 6657.740252] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 6657.878291] Key type .llcrypt unregistered [ 6657.880681] Key type ._llcrypt unregistered [ 6672.254308] Key type ._llcrypt registered [ 6672.256177] Key type .llcrypt registered [ 6672.307088] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 6672.458202] Key type .llcrypt unregistered [ 6672.460692] Key type ._llcrypt unregistered [ 6687.476618] Key type ._llcrypt registered [ 6687.478759] Key type .llcrypt registered [ 6687.549634] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 6687.681419] Key type .llcrypt unregistered [ 6687.683362] Key type ._llcrypt unregistered [ 6722.822069] Key type ._llcrypt registered [ 6722.823223] Key type .llcrypt registered [ 6722.870601] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 6723.016540] Key type .llcrypt unregistered [ 6723.018658] Key type ._llcrypt unregistered [ 6736.841361] Key type ._llcrypt registered [ 6736.842779] Key type .llcrypt registered [ 6736.883918] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 6737.011452] Key type .llcrypt unregistered [ 6737.013424] Key type ._llcrypt unregistered [ 6750.441485] Key type ._llcrypt registered [ 6750.443178] Key type .llcrypt registered [ 6750.491671] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 6750.658122] Key type .llcrypt unregistered [ 6750.660215] Key type ._llcrypt unregistered [ 6764.768246] Key type ._llcrypt registered [ 6764.770110] Key type .llcrypt registered [ 6764.832574] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 6764.986467] Key type .llcrypt unregistered [ 6764.988130] Key type ._llcrypt unregistered [ 6778.781687] Key type ._llcrypt registered [ 6778.783254] Key type .llcrypt registered [ 6778.841306] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 6778.980524] Key type .llcrypt unregistered [ 6778.982398] Key type ._llcrypt unregistered [ 6792.856128] Key type ._llcrypt registered [ 6792.857569] Key type .llcrypt registered [ 6792.895813] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 6793.000274] Key type .llcrypt unregistered [ 6793.002200] Key type ._llcrypt unregistered [ 6822.687645] Key type ._llcrypt registered [ 6822.689299] Key type .llcrypt registered [ 6822.750467] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 6822.925308] Key type .llcrypt unregistered [ 6822.927612] Key type ._llcrypt unregistered [ 6836.897128] Key type ._llcrypt registered [ 6836.898642] Key type .llcrypt registered [ 6836.942501] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 6837.060209] Key type .llcrypt unregistered [ 6837.061413] Key type ._llcrypt unregistered [ 6852.436414] Key type ._llcrypt registered [ 6852.437804] Key type .llcrypt registered [ 6852.483723] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 6852.616409] Key type .llcrypt unregistered [ 6852.618100] Key type ._llcrypt unregistered [ 6876.325709] Key type ._llcrypt registered [ 6876.326756] Key type .llcrypt registered [ 6876.370814] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 6884.553886] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 6884.877085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6884.894996] alg: No test for adler32 (adler32-zlib) [ 6885.749389] Lustre: Lustre: Build Version: 2.17.58_40_g357d090 [ 6885.834762] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 6887.415204] Key type lgssc registered [ 6887.954422] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6905.070405] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6905.083536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6906.188503] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 6906.202867] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 6906.243591] Lustre: tst250fs-MDT0000: new disk, initializing [ 6906.266787] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6906.307171] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6908.121108] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6913.720535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6913.793659] Lustre: 372721:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6913.837917] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 6913.841651] Lustre: Skipped 1 previous similar message [ 6913.887301] Lustre: tst250fs-MDT0001: new disk, initializing [ 6913.916947] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6913.927846] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6913.931407] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6915.628156] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6918.312567] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6921.550063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6921.729158] Lustre: tst250fs-OST0000: new disk, initializing [ 6921.732637] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 6921.737456] Lustre: 374354:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6921.785644] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6921.957431] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6921.962242] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6922.014068] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6924.426910] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6929.937400] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6930.036398] Lustre: tst250fs-OST0001: new disk, initializing [ 6930.040101] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 6930.046240] Lustre: 375225:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6930.104919] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6930.360339] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6930.366971] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6930.420060] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6932.783742] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6935.880525] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 6949.856578] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6949.861637] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6949.869943] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 6950.883985] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6950.885069] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 6950.891839] Lustre: Skipped 1 previous similar message [ 6950.895857] Lustre: Skipped 1 previous similar message [ 6956.002853] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 6956.007764] Lustre: Skipped 4 previous similar messages [ 6959.008684] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 6959.011848] Lustre: Skipped 2 previous similar messages [ 6962.143151] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6962.240172] Lustre: server umount tst250fs-MDT0000 complete [ 6963.774834] LustreError: 372713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789191826 with bad export cookie 9892910020789926601 [ 6963.780557] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6963.781603] LustreError: 372713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6963.924675] Lustre: server umount tst250fs-MDT0001 complete [ 6975.888770] Lustre: server umount tst250fs-OST0000 complete [ 6987.758396] Lustre: server umount tst250fs-OST0001 complete [ 6995.694960] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 6999.256297] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 7013.606140] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 01:44:35 (1789191875) [ 7014.377702] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 7021.040983] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 7024.702381] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 7039.732516] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6767 sec ========== 01:45:02 (1789191902) [ 7040.505389] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 01:45:02 (1789191902) === [ 7042.134843] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 01:45:04 (1789191904) === [ 7059.845264] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 7061.435641] Key type lgssc unregistered [ 7061.616745] LNet: 384347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7061.621576] LNetError: 384347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7061.636678] LNet: Removed LNI 192.168.206.145@tcp [ 7062.163192] Key type .llcrypt unregistered [ 7062.165323] Key type ._llcrypt unregistered