[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 529617684 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003345] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009010] pid_max: default: 32768 minimum: 301 [ 0.010145] LSM: Security Framework initializing [ 0.011049] Yama: becoming mindful. [ 0.012036] SELinux: Initializing. [ 0.013077] *** VALIDATE selinux *** [ 0.021813] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026184] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028096] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029091] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031075] *** VALIDATE tmpfs *** [ 0.033011] *** VALIDATE proc *** [ 0.034237] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.036280] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037152] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039031] Spectre V2 : User space: Vulnerable [ 0.040008] Speculative Store Bypass: Vulnerable [ 0.043180] debug: unmapping init [mem 0xffffffff9a859000-0xffffffff9a860fff] [ 0.045190] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046617] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047024] ... version: 2 [ 0.048012] ... bit width: 48 [ 0.049012] ... generic registers: 4 [ 0.050012] ... value mask: 0000ffffffffffff [ 0.051012] ... max period: 00007fffffffffff [ 0.052014] ... fixed-purpose events: 3 [ 0.053012] ... event mask: 000000070000000f [ 0.054301] rcu: Hierarchical SRCU implementation. [ 0.056371] smp: Bringing up secondary CPUs ... [ 0.057655] x86: Booting SMP configuration: [ 0.058027] .... node #0, CPUs: #1 #2 #3 [ 0.063015] smp: Brought up 1 node, 4 CPUs [ 0.065014] smpboot: Max logical packages: 1 [ 0.066010] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.149497] node 0 deferred pages initialised in 80ms [ 0.153135] devtmpfs: initialized [ 0.155254] x86/mm: Memory block size: 128MB [ 0.158967] gcov: version magic: 0x41383552 [ 0.161217] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.165108] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.168353] pinctrl core: initialized pinctrl subsystem [ 0.170159] [ 0.170716] ************************************************************* [ 0.174011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.176010] ** ** [ 0.179011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.181009] ** ** [ 0.183011] ** This means that this kernel is built to expose internal ** [ 0.185010] ** IOMMU data structures, which may compromise security on ** [ 0.188010] ** your system. ** [ 0.190009] ** ** [ 0.192012] ** If you see this message and you are not debugging the ** [ 0.195011] ** kernel, report this immediately to your vendor! ** [ 0.197008] ** ** [ 0.200010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.202009] ************************************************************* [ 0.204724] NET: Registered protocol family 16 [ 0.207397] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.210055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.213093] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.217087] cpuidle: using governor menu [ 0.218994] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.221489] PCI: Using configuration type 1 for base access [ 0.224134] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.234342] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.237075] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.242103] cryptd: max_cpu_qlen set to 1000 [ 0.245241] ACPI: Added _OSI(Module Device) [ 0.247011] ACPI: Added _OSI(Processor Device) [ 0.248011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.250012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.255318] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.262343] ACPI: Interpreter enabled [ 0.264143] ACPI: PM: (supports S0 S3 S4 S5) [ 0.266012] ACPI: Using IOAPIC for interrupt routing [ 0.268088] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.272365] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.284482] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.287038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.290016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.293131] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.299465] acpiphp: Slot [2] registered [ 0.301127] acpiphp: Slot [5] registered [ 0.302131] acpiphp: Slot [6] registered [ 0.304123] acpiphp: Slot [7] registered [ 0.306115] acpiphp: Slot [8] registered [ 0.308113] acpiphp: Slot [9] registered [ 0.309118] acpiphp: Slot [10] registered [ 0.311113] acpiphp: Slot [3] registered [ 0.313094] acpiphp: Slot [4] registered [ 0.315089] acpiphp: Slot [11] registered [ 0.317218] acpiphp: Slot [12] registered [ 0.320120] acpiphp: Slot [13] registered [ 0.323110] acpiphp: Slot [14] registered [ 0.324105] acpiphp: Slot [15] registered [ 0.326199] acpiphp: Slot [16] registered [ 0.328106] acpiphp: Slot [17] registered [ 0.329113] acpiphp: Slot [18] registered [ 0.331091] acpiphp: Slot [19] registered [ 0.333103] acpiphp: Slot [20] registered [ 0.335102] acpiphp: Slot [21] registered [ 0.336103] acpiphp: Slot [22] registered [ 0.338080] acpiphp: Slot [23] registered [ 0.340089] acpiphp: Slot [24] registered [ 0.341104] acpiphp: Slot [25] registered [ 0.343098] acpiphp: Slot [26] registered [ 0.345095] acpiphp: Slot [27] registered [ 0.347089] acpiphp: Slot [28] registered [ 0.349081] acpiphp: Slot [29] registered [ 0.351125] acpiphp: Slot [30] registered [ 0.352098] acpiphp: Slot [31] registered [ 0.354084] PCI host bridge to bus 0000:00 [ 0.356020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.358029] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.361023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.364024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.367021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.370035] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.372183] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.375195] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.379449] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.391014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.396893] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.400023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.402017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.405016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.408622] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.411827] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.415042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.419475] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.425013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.437903] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.442015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.449189] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.464017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.473013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.494015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.511036] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.518015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.526094] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.548014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.562102] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.575013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.589021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.623015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.641635] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.654014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.667022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.705015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.721226] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.739014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.753013] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.780015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.795594] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.806016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.813000] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.844016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.858324] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.860359] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.863375] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.866375] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.869236] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.873102] iommu: Default domain type: Passthrough [ 0.875385] SCSI subsystem initialized [ 0.877140] ACPI: bus type USB registered [ 0.879140] usbcore: registered new interface driver usbfs [ 0.881083] usbcore: registered new interface driver hub [ 0.884083] usbcore: registered new device driver usb [ 0.885173] pps_core: LinuxPPS API ver. 1 registered [ 0.888012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.891059] PTP clock support registered [ 0.894103] EDAC MC: Ver: 3.0.0 [ 0.896160] PCI: Using ACPI for IRQ routing [ 0.898896] NetLabel: Initializing [ 0.900015] NetLabel: domain hash size = 128 [ 0.901008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.904080] NetLabel: unlabeled traffic allowed by default [ 0.906274] vgaarb: loaded [ 0.907515] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.909012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.917000] clocksource: Switched to clocksource kvm-clock [ 1.027456] VFS: Disk quotas dquot_6.6.0 [ 1.031178] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.034635] *** VALIDATE ramfs *** [ 1.036320] *** VALIDATE hugetlbfs *** [ 1.038166] pnp: PnP ACPI init [ 1.040985] pnp: PnP ACPI: found 6 devices [ 1.057262] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.063704] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.066663] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.068430] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.070611] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.073120] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.076220] NET: Registered protocol family 2 [ 1.078699] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.083275] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.086930] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.091865] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.095231] TCP: Hash tables configured (established 65536 bind 65536) [ 1.099146] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.101749] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.104749] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.107119] NET: Registered protocol family 1 [ 1.109771] RPC: Registered named UNIX socket transport module. [ 1.112332] RPC: Registered udp transport module. [ 1.113992] RPC: Registered tcp transport module. [ 1.115762] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.118107] NET: Registered protocol family 44 [ 1.119888] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.124055] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.126261] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.128710] PCI: CLS 0 bytes, default 64 [ 1.130486] Unpacking initramfs... [ 2.571965] debug: unmapping init [mem 0xffff8a967cc54000-0xffff8a967ffbffff] [ 2.575933] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.578910] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.583861] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 3.088840] Initialise system trusted keyrings [ 3.090511] Key type blacklist registered [ 3.092908] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.103150] zbud: loaded [ 3.107130] *** VALIDATE nfs *** [ 3.108472] *** VALIDATE nfs4 *** [ 3.110101] pstore: using deflate compression [ 3.115105] Platform Keyring initialized [ 3.220560] NET: Registered protocol family 38 [ 3.226810] Key type asymmetric registered [ 3.228895] Asymmetric key parser 'x509' registered [ 3.231802] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.236472] io scheduler mq-deadline registered [ 3.238390] io scheduler kyber registered [ 3.240378] io scheduler bfq registered [ 3.242431] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.246325] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.250842] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.254853] ACPI: Power Button [PWRF] [ 3.261752] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.268554] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.285946] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.310172] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.334136] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.365417] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.394936] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.400622] Non-volatile memory driver v1.3 [ 3.402842] Linux agpgart interface v0.103 [ 3.441667] virtio_blk virtio1: [vda] 150408 512-byte logical blocks (77.0 MB/73.4 MiB) [ 3.445472] vda: detected capacity change from 0 to 77008896 [ 3.467493] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.471117] vdb: detected capacity change from 0 to 1073741824 [ 3.489735] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.492407] vdc: detected capacity change from 0 to 2621440000 [ 3.506154] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.509273] vdd: detected capacity change from 0 to 2621440000 [ 3.528100] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.531559] vde: detected capacity change from 0 to 4294967296 [ 3.544396] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.547296] vdf: detected capacity change from 0 to 4294967296 [ 3.555138] libphy: Fixed MDIO Bus: probed [ 3.591738] usbcore: registered new interface driver usbserial_generic [ 3.594718] usbserial: USB Serial support registered for generic [ 3.597424] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.601509] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.603375] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.605986] mousedev: PS/2 mouse device common for all mice [ 3.610838] rtc_cmos 00:05: RTC can wake from S4 [ 3.614918] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.616096] rtc_cmos 00:05: registered as rtc0 [ 3.625542] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.630541] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.631322] intel_pstate: CPU model not supported [ 3.636631] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.640650] hid: raw HID events driver (C) Jiri Kosina [ 3.643674] usbcore: registered new interface driver usbhid [ 3.645708] usbhid: USB HID core driver [ 3.647080] drop_monitor: Initializing network drop monitor service [ 3.649224] Initializing XFRM netlink socket [ 3.650854] NET: Registered protocol family 10 [ 3.654188] Segment Routing with IPv6 [ 3.655497] NET: Registered protocol family 17 [ 3.658135] mpls_gso: MPLS GSO support [ 3.663872] RAS: Correctable Errors collector initialized. [ 3.665823] AVX version of gcm_enc/dec engaged. [ 3.667737] AES CTR mode by8 optimization enabled [ 3.746482] sched_clock: Marking stable (3746445559, 0)->(4693259030, -946813471) [ 3.749722] registered taskstats version 1 [ 3.752631] Loading compiled-in X.509 certificates [ 3.755068] zswap: loaded using pool lzo/zbud [ 3.777980] Key type big_key registered [ 3.791362] Key type encrypted registered [ 3.792652] ima: No TPM chip found, activating TPM-bypass! [ 3.795906] ima: Allocated hash algorithm: sha1 [ 3.797258] ima: No architecture policies found [ 3.798652] evm: Initialising EVM extended attributes: [ 3.800207] evm: security.selinux [ 3.801279] evm: security.ima [ 3.802363] evm: security.capability [ 3.803607] evm: HMAC attrs: 0x1 [ 3.805825] rtc_cmos 00:05: setting system clock to 2026-09-22 17:22:45 UTC (1790097765) [ 3.812833] debug: unmapping init [mem 0xffffffff9b803000-0xffffffff9b9fffff] [ 3.816095] debug: unmapping init [mem 0xffffffff9a582000-0xffffffff9a858fff] [ 3.825061] Write protecting the kernel read-only data: 28672k [ 3.829400] debug: unmapping init [mem 0xffffffff98c03000-0xffffffff98dfffff] [ 3.832734] debug: unmapping init [mem 0xffffffff99514000-0xffffffff995fffff] [ 3.866708] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.875968] systemd[1]: Detected virtualization kvm. [ 3.877939] systemd[1]: Detected architecture x86-64. [ 3.880338] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.910108] systemd[1]: No hostname configured. [ 3.912025] systemd[1]: Set hostname to . [ 3.914450] random: systemd: uninitialized urandom read (16 bytes read) [ 3.917292] systemd[1]: Initializing machine ID from random generator. [ 3.972754] random: ln: uninitialized urandom read (6 bytes read) [ 4.068736] random: systemd: uninitialized urandom read (16 bytes read) [ 4.071819] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.077148] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.082770] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.747136] device-mapper: uevent: version 1.0.3 [ 4.749523] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.518471] virtio_net virtio0 ens2: renamed from eth0 [ 5.529067] random: fast init done [ 5.556295] scsi host0: ata_piix [ 5.588981] scsi host1: ata_piix [ 5.590752] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.593661] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.403292] dracut-initqueue[576]: RTNETLINK answers: File exists [ 10.308833] random: crng init done [ 10.310271] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.819973] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.965477] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.210492] SELinux: Disabled at runtime. [ 12.273187] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.280881] systemd[1]: Detected virtualization kvm. [ 12.282793] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.803194] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.806604] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.811822] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.818332] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.823865] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.830851] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.846492] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl s[ 13.044665] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS tatus systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 13.304384] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.615691] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.627526] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.711578] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.735260] EDAC sbridge: Ver: 1.1.2 [ 15.537126] Key type dns_resolver registered [ 15.846600] NFS: Registering the id_resolver key type [ 15.849112] Key type id_resolver registered [ 15.850886] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg351-server login: [ 44.576925] libcfs: loading out-of-tree module taints kernel. [ 44.666982] Key type ._llcrypt registered [ 44.669800] Key type .llcrypt registered [ 44.823304] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 63.345679] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 64.858840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 64.935195] alg: No test for adler32 (adler32-zlib) [ 66.381936] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 67.309690] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 69.103203] Key type lgssc registered [ 70.601275] Lustre: Echo OBD driver; http://www.lustre.org/ [ 85.475663] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 123.491710] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 135.844353] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 135.889638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 137.221430] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 137.268590] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 137.396882] Lustre: lustre-MDT0000: new disk, initializing [ 137.496658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 137.510765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 141.999215] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 147.060218] hrtimer: interrupt took 6490456 ns [ 153.537976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 153.601408] Lustre: 6497:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 153.630648] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 153.634332] Lustre: Skipped 1 previous similar message [ 153.693944] Lustre: lustre-MDT0001: new disk, initializing [ 153.763100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 153.797447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 153.811810] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 157.982946] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 162.942931] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 171.776805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 172.047781] Lustre: lustre-OST0000: new disk, initializing [ 172.052082] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 172.059410] Lustre: 8436:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 172.141798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 172.221252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 172.230699] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 172.387414] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 178.160177] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 192.443255] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 192.574919] Lustre: lustre-OST0001: new disk, initializing [ 192.579266] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 192.587949] Lustre: 9508:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 192.658218] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 198.925422] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 199.721495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 199.731778] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 199.831388] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 211.219693] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 216.618399] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 222.027206] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing check_logdir /tmp/testlogs/ [ 226.854802] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing yml_node [ 231.461860] Lustre: DEBUG MARKER: Client: 2.17.58.63 [ 234.076283] Lustre: DEBUG MARKER: MDS: 2.17.58.63 [ 236.345370] Lustre: DEBUG MARKER: OSS: 2.17.58.63 [ 237.803522] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 22 13:26:38 EDT 2026 [ 253.304382] Lustre: DEBUG MARKER: only running tests: 121-999 [ 254.622702] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 255.839969] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 271.347290] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 271.352486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 271.366756] Lustre: Skipped 1 previous similar message [ 271.382596] Lustre: Skipped 3 previous similar messages [ 276.516229] Lustre: server umount lustre-MDT0000 complete [ 279.638890] LustreError: 6490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790098041 with bad export cookie 17794479048493307584 [ 279.646624] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 281.569136] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 281.572025] LustreError: 6504:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 281.589023] Lustre: Skipped 2 previous similar messages [ 281.591564] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 281.612944] LustreError: 6504:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 281.640170] Lustre: Skipped 1 previous similar message [ 284.576406] LustreError: 11957:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 284.577913] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 284.591806] LustreError: 11957:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 284.613452] Lustre: Skipped 1 previous similar message [ 286.042458] Lustre: server umount lustre-MDT0001 complete [ 289.541087] Lustre: server umount lustre-OST0000 complete [ 293.084222] Lustre: server umount lustre-OST0001 complete [ 299.894363] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 309.764791] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 340.129551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 340.321568] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 340.344215] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 340.397581] Lustre: lustre-MDT0000: new disk, initializing [ 340.458183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 340.472423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 343.987275] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 352.240131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 352.354867] Lustre: 15104:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 352.421259] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 352.425848] Lustre: Skipped 1 previous similar message [ 352.503833] Lustre: lustre-MDT0001: new disk, initializing [ 352.583110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 352.611718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 352.628796] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 356.460809] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 364.535580] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 369.290667] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 374.619580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 374.815892] Lustre: lustre-OST0000: new disk, initializing [ 374.819566] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 374.824752] Lustre: 17072:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 374.882362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 376.371512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 376.379543] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 376.454190] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 379.093481] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 386.283520] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 390.048448] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 390.214205] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 393.568491] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 393.706444] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 396.771717] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 396.782270] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 396.796885] Lustre: Skipped 1 previous similar message [ 396.810268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 397.286748] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 401.526738] Lustre: server umount lustre-OST0000 complete [ 407.520790] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 407.528299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 407.537762] Lustre: Skipped 2 previous similar messages [ 407.548183] Lustre: Skipped 2 previous similar messages [ 410.828504] Lustre: server umount lustre-MDT0000 complete [ 412.642525] LustreError: 15115:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 412.667815] LustreError: 15115:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 413.671287] LustreError: 17071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790098175 with bad export cookie 17794479048493309124 [ 413.675968] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 413.681451] LustreError: 17071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 413.907333] Lustre: server umount lustre-MDT0001 complete [ 418.535863] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 13:29:39 (1790098179) [ 431.916458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 432.269431] LustreError: 19945:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 432.332601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 435.279720] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 437.730001] LustreError: 19946:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 441.239610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 444.296245] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 449.290969] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 453.393313] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 455.075868] Lustre: Failing over lustre-MDT0000 [ 455.300636] Lustre: server umount lustre-MDT0000 complete [ 457.187798] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 457.191870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 457.194617] LustreError: 20749:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 457.196427] Lustre: Skipped 1 previous similar message [ 471.056603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 471.143551] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 473.933720] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 476.644981] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 476.646125] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 476.666468] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 480.648442] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 485.860541] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 485.862283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 485.878690] Lustre: Skipped 1 previous similar message [ 485.880445] Lustre: Skipped 1 previous similar message [ 489.823836] Lustre: server umount lustre-MDT0000 complete [ 492.001320] LustreError: 19946:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 492.016122] LustreError: 19946:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 492.605517] LustreError: 19926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790098254 with bad export cookie 17794479048493310538 [ 492.615367] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 492.618035] LustreError: 19926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 492.830886] Lustre: server umount lustre-MDT0001 complete [ 499.138187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 499.492284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 499.500044] Lustre: Skipped 2 previous similar messages [ 502.427665] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 508.299592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 511.738838] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 516.803261] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 520.462466] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 521.904991] Lustre: Failing over lustre-MDT0000 [ 522.039861] Lustre: server umount lustre-MDT0000 complete [ 524.258104] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 524.260689] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 524.267875] LustreError: 23278:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 524.267886] LustreError: 23278:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 524.270469] Lustre: Skipped 1 previous similar message [ 539.327781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 539.425601] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 542.921332] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 544.745860] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 544.751805] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 544.759957] Lustre: Skipped 1 previous similar message [ 544.776436] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 549.426638] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 553.508033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 553.516394] Lustre: Skipped 3 previous similar messages [ 558.653556] Lustre: server umount lustre-MDT0000 complete [ 560.984173] LustreError: 24010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790098322 with bad export cookie 17794479048493311476 [ 560.984792] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 560.994513] LustreError: 24010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 561.200280] Lustre: server umount lustre-MDT0001 complete [ 566.182500] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 13:32:06 (1790098326) [ 576.385935] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 582.517954] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 602.437246] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 607.134816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 607.245213] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 607.259146] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 607.302972] Lustre: lustre-MDT0000: new disk, initializing [ 607.344401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 609.121740] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 614.834824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 614.904341] Lustre: 29639:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 615.095171] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 617.301272] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 622.943544] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 626.235870] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 630.023956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 630.192906] Lustre: 31606:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 630.239627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 630.243965] Lustre: Skipped 4 previous similar messages [ 632.139093] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 632.156192] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 632.202414] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 632.210636] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 633.952870] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 640.156974] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 645.363976] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 657.793776] Lustre: server umount lustre-OST0000 complete [ 661.472583] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 661.483130] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 661.495834] Lustre: Skipped 2 previous similar messages [ 661.501681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 661.505652] Lustre: Skipped 2 previous similar messages [ 670.265805] Lustre: server umount lustre-MDT0000 complete [ 672.622845] LustreError: 29631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790098434 with bad export cookie 17794479048493311882 [ 672.625511] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 672.632645] LustreError: 29631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 683.095988] Lustre: server umount lustre-MDT0001 complete [ 690.496169] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 692.335096] Key type lgssc unregistered [ 692.537612] LNet: 33562:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 692.542589] LNetError: 33562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 692.558090] LNet: Removed LNI 192.168.203.151@tcp [ 693.181309] Key type .llcrypt unregistered [ 693.183540] Key type ._llcrypt unregistered [ 709.954670] Key type ._llcrypt registered [ 709.957171] Key type .llcrypt registered [ 710.059360] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 710.649082] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 710.748597] alg: No test for adler32 (adler32-zlib) [ 711.743744] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 711.908915] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 713.567186] Key type lgssc registered [ 714.439333] Lustre: Echo OBD driver; http://www.lustre.org/ [ 720.403268] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 720.420288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 721.714430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 724.577388] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 730.255326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 730.516400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 733.371209] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 738.190128] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 741.749307] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 746.157793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 746.423499] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 750.524898] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 755.653414] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 755.783968] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 765.945558] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 13:35:26 (1790098526) [ 767.246194] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 768.408186] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 772.626268] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 13:35:33 (1790098533) [ 778.471846] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 13:35:39 (1790098539) [ 779.218586] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 785.643142] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 785.646486] Lustre: Skipped 2 previous similar messages [ 787.015522] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 13:35:47 (1790098547) [ 789.929980] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 789.935900] Lustre: Skipped 3 previous similar messages [ 802.942437] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 802.951680] Lustre: Skipped 6 previous similar messages [ 804.926853] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 13:36:05 (1790098565) [ 806.842170] Lustre: *** cfs_fail_loc=131b, val=2*** [ 809.382695] Lustre: *** cfs_fail_loc=131b, val=2*** [ 815.037244] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 13:36:15 (1790098575) [ 819.311746] Lustre: Disabling parameter general.jobid_name=TESTNAME1 in log params [ 819.317725] Lustre: Skipped 6 previous similar messages [ 825.440448] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 13:36:25 (1790098585) [ 837.604451] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 837.622988] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 837.631254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 840.675158] Lustre: server umount lustre-OST0000 complete [ 843.235557] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 843.239438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 843.245700] Lustre: Skipped 1 previous similar message [ 843.249888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 843.253406] Lustre: Skipped 1 previous similar message [ 847.844507] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 847.846298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 847.863513] Lustre: Skipped 1 previous similar message [ 849.288638] Lustre: server umount lustre-MDT0000 complete [ 851.967104] LustreError: 34952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790098613 with bad export cookie 7116627212960058786 [ 851.971834] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 851.976952] LustreError: 34952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 852.174431] Lustre: server umount lustre-MDT0001 complete [ 859.213733] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 861.091506] Key type lgssc unregistered [ 861.343623] LNet: 42809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 861.357402] LNetError: 42809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 861.376139] LNet: Removed LNI 192.168.203.151@tcp [ 861.988250] Key type .llcrypt unregistered [ 861.994149] Key type ._llcrypt unregistered [ 873.350308] Key type ._llcrypt registered [ 873.353303] Key type .llcrypt registered [ 873.439228] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 884.305245] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 884.931594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 884.943725] alg: No test for adler32 (adler32-zlib) [ 885.892416] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 886.102681] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 887.751208] Key type lgssc registered [ 888.501913] Lustre: Echo OBD driver; http://www.lustre.org/ [ 911.215630] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 911.235088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 912.396895] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 912.423760] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 912.496889] Lustre: lustre-MDT0000: new disk, initializing [ 912.558326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 912.574331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 915.157935] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 922.438853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 922.519676] Lustre: 45956:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 922.561850] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 922.564686] Lustre: Skipped 1 previous similar message [ 922.618341] Lustre: lustre-MDT0001: new disk, initializing [ 922.708167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 922.748114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 922.755797] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 925.621878] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 931.647682] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 935.123200] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 939.258577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 939.456111] Lustre: lustre-OST0000: new disk, initializing [ 939.459698] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 939.467390] Lustre: 47922:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 939.518484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 943.239299] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 948.250976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 948.260505] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 948.316213] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 949.505741] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 952.355482] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 952.466856] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 955.268542] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 955.414967] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 958.434555] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 958.435768] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 958.440366] LustreError: Skipped 1 previous similar message [ 958.453455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 958.462829] Lustre: Skipped 1 previous similar message [ 962.749755] Lustre: server umount lustre-OST0000 complete [ 968.673426] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 968.674702] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 968.681633] Lustre: Skipped 1 previous similar message [ 968.687401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 968.693671] Lustre: Skipped 1 previous similar message [ 971.306328] Lustre: server umount lustre-MDT0000 complete [ 973.593313] LustreError: 45947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790098735 with bad export cookie 1075652989718803690 [ 973.594833] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 973.599989] LustreError: 45947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 973.824065] Lustre: server umount lustre-MDT0001 complete [ 979.801150] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 13:39:00 (1790098740) [ 983.972474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 984.279975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 986.939562] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 989.665126] LustreError: 50380:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 989.681094] LustreError: 50380:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 992.406911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 992.624250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 995.443241] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1000.505937] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1004.400833] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1009.466762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1009.755780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1014.286822] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1019.204162] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1032.434795] LustreError: 52930:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1032.933672] Lustre: Modifying parameter general.timeout=1 in log params [ 1034.373430] Lustre: Modifying parameter general.timeout=4 in log params [ 1034.379057] Lustre: Skipped 2 previous similar messages [ 1036.874683] Lustre: Modifying parameter general.timeout=9 in log params [ 1036.878828] Lustre: Skipped 4 previous similar messages [ 1041.386141] Lustre: Modifying parameter general.timeout=17 in log params [ 1041.390740] Lustre: Skipped 7 previous similar messages [ 1049.507903] Lustre: Modifying parameter general.timeout=31 in log params [ 1049.513427] Lustre: Skipped 13 previous similar messages [ 1065.949143] Lustre: Modifying parameter general.timeout=57 in log params [ 1065.951937] Lustre: Skipped 25 previous similar messages [ 1098.030553] Lustre: Modifying parameter general.timeout=106 in log params [ 1098.034971] Lustre: Skipped 48 previous similar messages [ 1119.015200] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 13:41:19 (1790098879) [ 1121.070751] Lustre: 59847:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1122.257153] Lustre: 59946:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1124.792526] Lustre: 60158:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1124.799403] Lustre: 60158:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1144.722530] Lustre: 61702:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1144.731843] Lustre: 61702:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1144.745533] LustreError: 61702:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1147.773759] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 13:41:48 (1790098908) [ 1158.119577] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1158.121481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1158.134047] Lustre: Skipped 2 previous similar messages [ 1158.138124] Lustre: Skipped 1 previous similar message [ 1163.236657] LustreError: 50380:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1163.254688] LustreError: 50380:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1163.300746] Lustre: server umount lustre-MDT0000 complete [ 1165.675187] LustreError: 53916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790098927 with bad export cookie 1075652989718804572 [ 1165.682672] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1165.684570] LustreError: 53916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1165.918851] Lustre: server umount lustre-MDT0001 complete [ 1178.981960] Lustre: server umount lustre-OST0000 complete [ 1196.159531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1196.337686] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1196.371424] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1196.377794] Lustre: Skipped 46 previous similar messages [ 1196.544247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1200.559322] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1209.403124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1209.505014] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1209.526937] Lustre: 64230:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1213.242298] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1218.478418] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1221.579328] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1225.457175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1225.558690] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1229.032234] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1232.721082] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1238.772111] Lustre: 66798:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1245.157107] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1245.163703] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1245.171069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1245.173446] Lustre: Skipped 2 previous similar messages [ 1248.040884] Lustre: server umount lustre-OST0000 complete [ 1250.273179] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1250.274987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1250.280626] Lustre: Skipped 2 previous similar messages [ 1256.108570] Lustre: server umount lustre-MDT0000 complete [ 1257.889343] LustreError: 64222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790099019 with bad export cookie 1075652989718807239 [ 1257.892298] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1257.895288] LustreError: 64222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1258.063662] Lustre: server umount lustre-MDT0001 complete [ 1263.316465] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 1264.722131] Key type lgssc unregistered [ 1264.891108] LNet: 68004:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1264.896883] LNetError: 68004:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1264.912689] LNet: Removed LNI 192.168.203.151@tcp [ 1265.375810] Key type .llcrypt unregistered [ 1265.378591] Key type ._llcrypt unregistered [ 1279.134906] Key type ._llcrypt registered [ 1279.136546] Key type .llcrypt registered [ 1279.192772] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 1279.787984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1279.804198] alg: No test for adler32 (adler32-zlib) [ 1280.755913] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 1280.902652] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 1282.543164] Key type lgssc registered [ 1283.202497] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1287.961526] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1287.974723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1289.266783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1291.743958] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1296.490398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1296.822648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1299.364586] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1303.782369] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1307.220437] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1311.400826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1311.724895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1315.080972] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1319.376586] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1329.561287] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1334.774656] Lustre: server umount lustre-MDT0000 complete [ 1337.034351] LustreError: 71262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790099098 with bad export cookie 5785268998703780195 [ 1337.041817] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1337.042894] LustreError: 71262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1337.313681] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1337.319227] LustreError: 69406:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1337.324686] Lustre: Skipped 1 previous similar message [ 1337.326208] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1337.352461] LustreError: 69406:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1339.289778] Lustre: server umount lustre-MDT0001 complete [ 1360.438850] Lustre: server umount lustre-OST0000 complete [ 1374.214834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1374.359637] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1374.385277] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1374.547916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1376.693970] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1381.105851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1381.157674] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1381.173630] Lustre: 74277:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1381.381047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1383.329277] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1387.034869] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1389.878968] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1393.252528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1393.350077] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1393.564382] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1396.733555] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1400.346989] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1401.043190] Lustre: 76849:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1401.859729] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1407.969983] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1407.970136] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1407.973267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1407.975962] LustreError: Skipped 1 previous similar message [ 1407.990360] Lustre: Skipped 1 previous similar message [ 1413.089758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1413.093880] Lustre: Skipped 2 previous similar messages [ 1418.207133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1418.209804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1418.217062] Lustre: Skipped 1 previous similar message [ 1418.297785] Lustre: server umount lustre-OST0000 complete [ 1422.304125] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1422.308757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1422.315501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1426.380490] Lustre: server umount lustre-MDT0000 complete [ 1428.294792] LustreError: 75022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790099189 with bad export cookie 5785268998703781007 [ 1428.299722] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1428.301055] LustreError: 75022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1428.443558] Lustre: server umount lustre-MDT0001 complete [ 1433.709871] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 1435.363816] Key type lgssc unregistered [ 1435.535610] LNet: 78055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1435.541821] LNetError: 78055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1435.553786] LNet: Removed LNI 192.168.203.151@tcp [ 1435.976480] Key type .llcrypt unregistered [ 1435.978603] Key type ._llcrypt unregistered [ 1456.400488] Key type ._llcrypt registered [ 1456.402893] Key type .llcrypt registered [ 1456.482901] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 1457.580462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1457.620727] alg: No test for adler32 (adler32-zlib) [ 1459.011932] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 1459.463531] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 1461.418088] Key type lgssc registered [ 1463.346864] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1471.013687] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1471.033723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1472.755327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1478.710094] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1483.234541] LustreError: 79482:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1483.248600] LustreError: 79482:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1487.665745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1488.217773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1493.082607] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1499.596585] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1504.083673] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1509.115099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1509.612490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1514.264662] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1519.726898] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1525.065063] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1525.624579] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1526.829269] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1526.832163] Lustre: Skipped 1 previous similar message [ 1529.448685] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1529.453506] Lustre: Skipped 3 previous similar messages [ 1534.014935] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1534.018885] Lustre: Skipped 6 previous similar messages [ 1542.590954] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1542.594549] Lustre: Skipped 11 previous similar messages [ 1559.149923] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1559.152709] Lustre: Skipped 23 previous similar messages [ 1591.266877] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1591.272363] Lustre: Skipped 46 previous similar messages [ 1598.125247] Lustre: 87568:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1599.936665] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 13:49:20 (1790099360) [ 1601.109501] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1602.308526] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 13:49:22 (1790099362) [ 1605.089123] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1605.090903] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1605.096819] LustreError: Skipped 1 previous similar message [ 1605.097674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1605.123781] Lustre: Skipped 1 previous similar message [ 1610.211317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1610.215850] Lustre: Skipped 1 previous similar message [ 1610.812455] Lustre: server umount lustre-OST0000 complete [ 1615.331049] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1615.335194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1615.346920] Lustre: Skipped 1 previous similar message [ 1615.356701] Lustre: Skipped 2 previous similar messages [ 1619.316545] Lustre: server umount lustre-MDT0000 complete [ 1620.448742] LustreError: 79478:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1620.460175] LustreError: 79478:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1622.281229] LustreError: 83254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790099383 with bad export cookie 2112752820550993368 [ 1622.284860] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1622.296276] LustreError: 83254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1622.560637] Lustre: server umount lustre-MDT0001 complete [ 1629.932918] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 1632.009977] Key type lgssc unregistered [ 1632.222749] LNet: 88816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1632.240084] LNetError: 88816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1633.253748] LNet: Removed LNI 192.168.203.151@tcp [ 1633.961362] Key type .llcrypt unregistered [ 1633.964187] Key type ._llcrypt unregistered [ 1637.578350] Key type ._llcrypt registered [ 1637.581555] Key type .llcrypt registered [ 1637.669352] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1641.108035] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules [ 1642.013130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1642.038415] alg: No test for adler32 (adler32-zlib) [ 1643.020726] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 1643.245631] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 1644.905513] Key type lgssc registered [ 1646.011192] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1646.991392] LustreError: 90033:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 1660.769389] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 1668.181062] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1668.431075] LustreError: 90033:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1668.460537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1669.871496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1673.390536] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1679.334518] LustreError: 91058:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1679.353324] LustreError: 91058:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1680.386145] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 13:50:40 (1790099440) [ 1684.184389] LustreError: 91089:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1684.197513] LustreError: 91089:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1684.212665] LustreError: 91089:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 14, retries 0, failed: rc = -5 [ 1684.442328] Lustre: server umount lustre-MDT0000 complete [ 1694.215960] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 1696.149249] Key type lgssc unregistered [ 1696.348645] LNet: 92580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1696.354435] LNetError: 92580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1696.364941] LNet: Removed LNI 192.168.203.151@tcp [ 1696.892657] Key type .llcrypt unregistered [ 1696.894638] Key type ._llcrypt unregistered [ 1708.939091] Key type ._llcrypt registered [ 1708.941314] Key type .llcrypt registered [ 1709.003959] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 1709.676972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1709.698505] alg: No test for adler32 (adler32-zlib) [ 1710.666888] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 1710.808384] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 1712.431952] Key type lgssc registered [ 1713.164428] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1718.586751] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1718.600814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1719.918234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1723.346725] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1729.505813] LustreError: 93790:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1729.526776] LustreError: 93790:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1730.734105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1731.319071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1735.390578] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1741.481571] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1745.852756] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1750.709876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1751.297853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1755.612528] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1761.741220] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1805.182498] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 13:52:45 (1790099565) [ 1806.509480] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1807.709171] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 13:52:48 (1790099568) [ 1810.915426] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1810.927105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1810.928136] Lustre: Skipped 1 previous similar message [ 1816.033476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1816.049131] Lustre: Skipped 4 previous similar messages [ 1816.885105] Lustre: server umount lustre-MDT0000 complete [ 1820.244026] LustreError: 94531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790099581 with bad export cookie 12767621671172079288 [ 1820.252162] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1820.259160] LustreError: 94531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1820.574957] Lustre: server umount lustre-MDT0001 complete [ 1834.174472] Lustre: server umount lustre-OST0000 complete [ 1845.054370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1845.315886] LustreError: 98063:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1845.435752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1848.981625] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1850.852527] LustreError: 98064:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1855.578584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1856.115532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1859.477755] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1864.436361] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1868.592433] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1878.107945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1878.241527] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1878.247426] LustreError: 98053:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1878.264605] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1878.271176] LustreError: 100146:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 1878.275568] LustreError: 100146:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 1878.279810] LustreError: 100146:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1878.349582] Lustre: server umount lustre-OST0000 complete [ 1878.353943] LustreError: 100146:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1884.469831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1884.612405] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1884.618369] LustreError: 98053:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1884.631954] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1884.642794] LustreError: 100607:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 1884.650575] LustreError: 100607:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 1884.656235] LustreError: 100607:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1884.724504] Lustre: server umount lustre-OST0000 complete [ 1884.728799] LustreError: 100607:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1891.394472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1891.505497] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1891.508579] Lustre: Found index 0 for lustre-OST0000, updating log [ 1891.512634] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1891.563646] Lustre: lustre-OST0000: new disk, initializing [ 1891.566585] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1891.571813] Lustre: 101141:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1891.743427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1895.842650] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1898.357494] Lustre: Failing over lustre-OST0000 [ 1898.496903] Lustre: server umount lustre-OST0000 complete [ 1902.050255] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1902.062636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1902.079566] Lustre: Skipped 1 previous similar message [ 1902.083882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1905.633653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1905.637908] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1910.753610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1910.767486] Lustre: Skipped 2 previous similar messages [ 1915.359124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1915.468644] Lustre: server umount lustre-MDT0000 complete [ 1915.872231] LustreError: 98795:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1915.884350] LustreError: 98795:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1917.776720] LustreError: 98043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790099679 with bad export cookie 12767621671172082032 [ 1917.783068] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1917.789636] LustreError: 98043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1918.014609] Lustre: server umount lustre-MDT0001 complete [ 1923.377578] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 13:54:44 (1790099684) [ 1932.974105] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 1939.422921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1939.683659] LustreError: 103488:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1939.765924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1942.563582] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1947.702258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1950.924797] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1952.767703] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1957.068223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1957.524603] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 1958.565262] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1958.573157] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 1958.582052] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 1961.409526] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1964.004646] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 1964.018671] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 1965.027977] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:03 [ 1965.040638] Lustre: lustre-OST0000: Recovery over after 0:07, of 1 clients 1 recovered and 0 were evicted. [ 1965.040774] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 1965.055870] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 1966.543045] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1966.595453] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1966.666434] Lustre: lustre-OST0001: new disk, initializing [ 1966.672465] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1966.680770] Lustre: 105807:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1967.938622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1967.948395] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1968.018568] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1970.631502] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1976.207821] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1981.673777] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1986.016382] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 1986.027533] LustreError: Skipped 1 previous similar message [ 1986.030670] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1986.038750] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1990.608649] Lustre: server umount lustre-MDT0001 complete [ 1992.899473] LustreError: 103485:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1992.908463] LustreError: 103485:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1996.054963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1996.118898] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1996.123330] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1996.152696] Lustre: 103477:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1996.596030] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1996.601936] Lustre: Skipped 3 previous similar messages [ 1997.607723] LustreError: 107402:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1997.621522] LustreError: 107402:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1997.628686] LustreError: 103480:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1997.696134] LustreError: 107417:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.151@tcp: cfg command failed: rc = -17 [ 1997.705976] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1997.705976] [ 1998.931207] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2001.898959] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2004.016543] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 13:56:04 (1790099764) [ 2013.615983] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 2024.646868] Lustre: 109415:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2039.187664] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2041.266890] Lustre: 110549:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2108.898507] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2108.904292] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2108.910900] Lustre: Skipped 2 previous similar messages [ 2108.913527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2108.916367] Lustre: Skipped 4 previous similar messages [ 2112.522828] Lustre: server umount lustre-MDT0000 complete [ 2114.529555] LustreError: 103489:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2114.543601] LustreError: 103489:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 2114.563378] LustreError: 103469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790099876 with bad export cookie 12767621671172082963 [ 2114.564690] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2114.575203] LustreError: 103469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2114.759847] Lustre: server umount lustre-MDT0001 complete [ 2126.899690] Lustre: server umount lustre-OST0000 complete [ 2138.567767] Lustre: server umount lustre-OST0001 complete [ 2141.926537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2147.167841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2151.779752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2157.673321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2169.648274] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 2175.464695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2175.478874] Lustre: lustre-MDT0000: reset Object Index mappings [ 2175.774620] LustreError: 114491:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2175.826924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2178.401840] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2183.739140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2183.754263] Lustre: lustre-MDT0001: reset Object Index mappings [ 2186.528556] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2188.170289] Lustre: 115639:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2191.825603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2194.286870] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:225) [ 2195.387657] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2200.244283] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2200.555555] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 2204.412799] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2206.195107] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 2206.198180] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 2209.380910] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2211.477239] Lustre: 117513:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2240.040442] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 14:00:00 (1790100000) [ 2271.200628] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2271.204040] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2271.211669] Lustre: Skipped 3 previous similar messages [ 2271.215905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2271.219158] Lustre: Skipped 4 previous similar messages [ 2274.369973] Lustre: server umount lustre-MDT0000 complete [ 2276.383405] LustreError: 115244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790100038 with bad export cookie 12767621671172130017 [ 2276.386362] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2276.392972] LustreError: 115244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2277.858418] LustreError: 114486:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2277.867897] LustreError: 114486:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2282.725066] Lustre: server umount lustre-MDT0001 complete [ 2299.359137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2299.430985] Lustre: server umount lustre-OST0000 complete [ 2307.599523] Lustre: server umount lustre-OST0001 complete [ 2310.979160] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 2316.186034] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 2348.100369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2348.305499] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2348.329486] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2348.465494] Lustre: lustre-MDT0000: new disk, initializing [ 2348.700520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2348.711410] Lustre: Skipped 3 previous similar messages [ 2348.743607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2354.383346] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2362.225332] Lustre: server umount lustre-MDT0000 complete [ 2373.972542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2374.451273] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2374.468627] Lustre: 121786:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2374.507293] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2378.970917] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2383.283271] Lustre: server umount lustre-MDT0000 complete [ 2392.339390] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 14:02:31 (1790100151) [ 2394.110893] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2395.928662] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 14:02:36 (1790100156) [ 2415.090193] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 2425.739860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2429.838816] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2436.601068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2436.673079] Lustre: 123837:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2436.709167] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2436.712251] Lustre: Skipped 1 previous similar message [ 2436.749156] Lustre: lustre-MDT0001: new disk, initializing [ 2436.825960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2436.834878] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2439.378995] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2442.751867] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2446.708862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2446.870686] Lustre: lustre-OST0000: new disk, initializing [ 2446.873530] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2446.877925] Lustre: 125417:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2448.220253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2448.232896] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2448.297168] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2451.089368] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2458.750574] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2458.887911] Lustre: lustre-OST0001: new disk, initializing [ 2458.891904] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2458.897774] Lustre: 126288:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2460.942985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2460.952550] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2461.012221] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2463.402906] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2471.130866] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2473.776764] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2840.413940] Lustre: 132695:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 2840.422823] Lustre: 132695:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 2840.432264] Lustre: 132695:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 2840.440252] Lustre: 132695:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 2840.447711] Lustre: 132695:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 2840.458691] Lustre: 132695:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3159.366509] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 14:15:18 (1790100918) [ 3174.881126] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3174.893069] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3174.912173] Lustre: Skipped 5 previous similar messages [ 3174.917756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3174.920633] Lustre: Skipped 8 previous similar messages [ 3177.953859] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3177.975347] Lustre: Skipped 1 previous similar message [ 3181.240668] Lustre: server umount lustre-MDT0000 complete [ 3183.074629] LustreError: 127101:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3183.094869] LustreError: 127101:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 3185.354824] LustreError: 124580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790100947 with bad export cookie 12767621671172191379 [ 3185.369642] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3185.379764] LustreError: 124580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3188.195830] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3188.207836] Lustre: Skipped 1 previous similar message [ 3188.219725] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3188.230104] Lustre: Skipped 5 previous similar messages [ 3192.102770] Lustre: server umount lustre-MDT0001 complete [ 3202.836540] Lustre: server umount lustre-OST0000 complete [ 3213.758529] Lustre: server umount lustre-OST0001 complete [ 3222.648979] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 3235.605400] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 3279.221794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3279.366202] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3279.397263] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3279.455593] Lustre: lustre-MDT0000: new disk, initializing [ 3279.518805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3279.523915] Lustre: Skipped 5 previous similar messages [ 3279.548996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3283.758666] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3296.684281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3296.961743] Lustre: 135752:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3297.078629] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3297.087118] Lustre: Skipped 1 previous similar message [ 3297.185862] Lustre: lustre-MDT0001: new disk, initializing [ 3297.398764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3297.420524] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3302.929338] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3314.498396] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3321.141414] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3329.400201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3329.953163] Lustre: lustre-OST0000: new disk, initializing [ 3329.958494] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3329.977419] Lustre: 137722:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3330.182881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3330.190134] Lustre: Skipped 1 previous similar message [ 3330.972951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3330.979644] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3331.124616] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3337.621843] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3347.907271] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3358.842754] Lustre: lustre-MDD0000: changelog on [ 3361.995052] Lustre: lustre-MDD0001: changelog on [ 5489.022985] Lustre: lustre-MDD0001: changelog off [ 5490.865309] Lustre: 139166:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5491.873740] Lustre: lustre-MDD0000: changelog off [ 5501.927123] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5501.934943] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5501.938065] LustreError: Skipped 1 previous similar message [ 5501.954418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5501.960496] Lustre: Skipped 2 previous similar messages [ 5501.969570] Lustre: Skipped 2 previous similar messages [ 5507.045561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5507.069808] Lustre: Skipped 2 previous similar messages [ 5512.169709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5512.184048] Lustre: Skipped 1 previous similar message [ 5513.183184] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5513.374151] Lustre: server umount lustre-OST0000 complete [ 5519.339115] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5519.355505] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5522.403927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5522.406935] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5522.408934] Lustre: Skipped 1 previous similar message [ 5524.574204] Lustre: server umount lustre-MDT0000 complete [ 5527.523381] LustreError: 139166:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5527.547260] LustreError: 139166:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 5528.495319] LustreError: 136548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790103290 with bad export cookie 12767621671173005437 [ 5528.499255] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5528.505851] LustreError: 136548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5529.133931] Lustre: server umount lustre-MDT0001 complete [ 5541.622694] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 5544.164160] Key type lgssc unregistered [ 5544.506183] LNet: 146899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5544.512547] LNetError: 146899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5544.544583] LNet: Removed LNI 192.168.203.151@tcp [ 5545.547593] Key type .llcrypt unregistered [ 5545.553683] Key type ._llcrypt unregistered [ 5570.199841] Key type ._llcrypt registered [ 5570.202556] Key type .llcrypt registered [ 5570.325304] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 5590.514662] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 5591.768272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5591.804713] alg: No test for adler32 (adler32-zlib) [ 5593.078337] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 5593.653590] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 5595.479411] Key type lgssc registered [ 5597.152361] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5636.980483] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5637.018129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5638.371462] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5638.434873] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5638.623277] Lustre: lustre-MDT0000: new disk, initializing [ 5638.696447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5638.723085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5643.295891] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 5654.001824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5654.136750] Lustre: 150135:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5654.239929] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5654.243253] Lustre: Skipped 1 previous similar message [ 5654.321374] Lustre: lustre-MDT0001: new disk, initializing [ 5654.441238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5654.481920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5654.502107] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5659.177662] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 5673.992359] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5683.067155] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5694.214682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5694.772277] Lustre: lustre-OST0000: new disk, initializing [ 5694.783782] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5694.802868] Lustre: 152103:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5694.934770] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5703.031205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5703.059725] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5703.676545] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5707.254198] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 5719.959552] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5726.566610] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_module obdecho/obdecho [ 5728.756903] LustreError: 153108:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5728.772779] LustreError: 153108:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 5731.208027] LustreError: 153205:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 5746.660092] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5746.670764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5746.697189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5749.767673] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5749.768733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5749.783659] Lustre: Skipped 1 previous similar message [ 5749.805068] Lustre: Skipped 2 previous similar messages [ 5751.931534] Lustre: server umount lustre-MDT0000 complete [ 5756.514413] LustreError: 150127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790103518 with bad export cookie 12717338139409641807 [ 5756.525080] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5757.083186] Lustre: server umount lustre-MDT0001 complete [ 5771.580539] Lustre: server umount lustre-OST0000 complete [ 5794.103280] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 5804.297527] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 5842.795713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5843.044936] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5843.087442] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5843.195391] Lustre: lustre-MDT0000: new disk, initializing [ 5843.334387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5843.370231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5848.375972] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 5857.758661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5857.889089] Lustre: 156923:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5857.961091] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5857.969975] Lustre: Skipped 1 previous similar message [ 5858.059325] Lustre: lustre-MDT0001: new disk, initializing [ 5858.144539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5858.179705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5858.191468] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5861.649220] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 5870.587306] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5876.555329] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5882.931050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5883.165754] Lustre: lustre-OST0000: new disk, initializing [ 5883.170121] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5883.176887] Lustre: 158892:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5883.265988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5889.251676] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 5891.141895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5891.148949] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5891.229902] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5897.645509] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5902.566742] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5902.791170] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5907.398938] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5907.601681] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5912.040645] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5912.043781] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5912.053277] LustreError: Skipped 1 previous similar message [ 5912.062862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5915.560393] Lustre: server umount lustre-OST0000 complete [ 5919.716027] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5919.727387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5919.752471] Lustre: Skipped 1 previous similar message [ 5919.762910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5919.771750] Lustre: Skipped 1 previous similar message [ 5925.135194] Lustre: server umount lustre-MDT0000 complete [ 5927.400694] LustreError: 158900:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5927.417979] LustreError: 158900:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 5928.286484] LustreError: 156914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790103689 with bad export cookie 12717338139409642766 [ 5928.290354] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5928.302742] LustreError: 156914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5928.600390] Lustre: server umount lustre-MDT0001 complete [ 5931.676463] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 15:01:31 (1790103691) [ 5933.212151] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 5935.188611] Lustre: DEBUG MARKER: == conf-sanity test 138: MDT mount failure must not LBUG destroying recovery exports ========================================================== 15:01:35 (1790103695) [ 5951.278856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5951.916932] LustreError: 161876:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5952.021290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5956.715586] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 5957.094041] LustreError: 161877:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5957.143968] LustreError: 161877:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5962.214835] LustreError: 161876:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5964.655256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5969.216437] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 5975.900162] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5980.988599] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5986.398130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5986.674737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5986.680158] Lustre: Skipped 1 previous similar message [ 5991.457651] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 5998.095894] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6002.628959] Lustre: *** cfs_fail_loc=718, val=0*** [ 6003.952995] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.51@tcp (stopping) [ 6003.968358] Lustre: Skipped 2 previous similar messages [ 6004.923992] Lustre: server umount lustre-MDT0000 complete [ 6006.241199] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6006.257979] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6006.282062] Lustre: Skipped 1 previous similar message [ 6006.292673] LustreError: 161877:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6008.961734] LustreError: 161858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790103770 with bad export cookie 12717338139409643648 [ 6008.967967] Lustre: *** cfs_fail_loc=718, val=0*** [ 6008.976630] LustreError: 161858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6008.976946] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6008.979567] Lustre: Skipped 3 previous similar messages [ 6009.390862] Lustre: server umount lustre-MDT0001 complete [ 6018.886833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6028.780655] Lustre: 148346:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790103774/real 1790103774] req@000000004d857895 x1877059415689600/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790103790 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6028.814904] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6028.826770] Lustre: Skipped 2 previous similar messages [ 6029.793806] Lustre: 148347:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790103775/real 1790103775] req@00000000d3bb8605 x1877059415689728/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790103791 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6033.887213] Lustre: 148348:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790103779/real 1790103779] req@00000000bf520386 x1877059415689856/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790103795 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6033.952309] LustreError: 148345:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006510e4c1 x1877059415690752/t0(0) o250->MGC192.168.203.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6034.224615] Lustre: *** cfs_fail_loc=723, val=0*** [ 6034.227372] Lustre: Skipped 1 previous similar message [ 6034.289698] LustreError: 165230:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-122) [ 6034.295228] LustreError: 165230:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.151@tcp: cfg command failed: rc = -122 [ 6034.299803] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 6034.299803] [ 6034.308345] LustreError: MGC192.168.203.151@tcp: Confguration from log lustre-MDT0000 failed from MGS -122. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 6034.319042] LustreError: 165200:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -122 [ 6034.326310] LustreError: 165200:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -122 [ 6034.333827] LustreError: 165200:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 6034.339796] LustreError: 165200:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 6034.415376] Lustre: server umount lustre-MDT0000 complete [ 6034.421771] LustreError: 165200:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -122 [ 6039.968223] Lustre: 148346:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790103785/real 1790103785] req@00000000d63df813 x1877059415690240/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790103801 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6040.000609] Lustre: 148346:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6040.800948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6040.933484] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6040.947918] LustreError: 161869:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 6041.144216] LustreError: 163705:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6041.163099] LustreError: 163705:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 6044.297152] Lustre: 148347:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790103789/real 1790103789] req@00000000abda5e18 x1877059415690496/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790103805 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6045.103055] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6047.338872] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6052.500853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6052.764517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6052.771575] Lustre: Skipped 1 previous similar message [ 6052.815048] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 6053.577853] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6053.812580] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6053.826334] Lustre: lustre-MDT0001: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 6056.479976] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6062.785767] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6064.009864] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6068.257903] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6069.583494] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6078.435565] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6078.442142] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6078.458645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6078.462560] Lustre: Skipped 1 previous similar message [ 6083.936809] Lustre: server umount lustre-OST0000 complete [ 6088.674846] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6093.190874] Lustre: server umount lustre-MDT0000 complete [ 6094.816570] LustreError: 165642:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6094.828885] LustreError: 165642:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 6095.912512] LustreError: 161859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790103857 with bad export cookie 12717338139409644600 [ 6095.917391] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6095.925360] LustreError: 161859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6096.146660] Lustre: server umount lustre-MDT0001 complete [ 6104.475245] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 6106.535265] Key type lgssc unregistered [ 6106.762769] LNet: 168456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6106.769130] LNetError: 168456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6107.814289] LNet: Removed LNI 192.168.203.151@tcp [ 6108.435547] Key type .llcrypt unregistered [ 6108.437414] Key type ._llcrypt unregistered [ 6122.461511] Key type ._llcrypt registered [ 6122.467515] Key type .llcrypt registered [ 6122.546595] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 6123.341262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6123.398266] alg: No test for adler32 (adler32-zlib) [ 6124.375158] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 6124.577357] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 6126.239415] Key type lgssc registered [ 6127.088889] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6132.899606] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6132.935471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6134.332026] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6138.149926] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6143.460293] LustreError: 169767:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6143.478544] LustreError: 169767:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6145.040116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6145.275899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6148.780611] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6155.378195] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6160.175366] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6165.218632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6165.489164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6170.406409] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6175.964654] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6178.805425] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6187.738962] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.51@tcp (stopping) [ 6189.029454] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6189.033669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6189.048538] Lustre: Skipped 1 previous similar message [ 6189.061359] Lustre: Skipped 2 previous similar messages [ 6192.295465] Lustre: server umount lustre-MDT0000 complete [ 6192.833305] LustreError: 170863:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6194.144114] LustreError: 169767:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6194.153693] LustreError: 169767:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6194.866901] LustreError: 169748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790103956 with bad export cookie 11864984046420349867 [ 6194.879113] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6197.959395] LustreError: 169762:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6197.976534] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.51@tcp (stopping) [ 6199.271077] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6199.286101] Lustre: Skipped 1 previous similar message [ 6201.185205] Lustre: server umount lustre-MDT0001 complete [ 6206.699881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6229.200590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6229.449884] LustreError: 173608:0:(llog.c:1655:llog_backup()) MGC192.168.203.151@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6229.454503] Lustre: 173608:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.151@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6229.739887] LustreError: 173613:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6229.776054] LustreError: 173613:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6238.016314] LustreError: 173613:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6238.026120] LustreError: 173613:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6256.096976] LustreError: 173615:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6256.125220] LustreError: 173615:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 16 previous similar messages [ 6289.605984] LustreError: 173614:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6289.618504] LustreError: 173614:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 22 previous similar messages [ 6290.711327] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6290.724536] Lustre: 173608:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.151@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6331.392386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6335.627587] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6336.500750] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6344.076273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6355.142983] LustreError: 173613:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6355.176263] LustreError: 173613:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 43 previous similar messages [ 6366.176031] LustreError: 169065:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b6c9514c x1877059972414208/t0(0) o250->MGC192.168.203.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6366.214841] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 6366.609480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6366.693955] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6370.850895] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6371.828950] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6380.261903] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6382.065949] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6387.690095] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6389.690095] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6412.255146] Lustre: server umount lustre-MDT0000 complete [ 6412.768724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6412.773364] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6412.821418] Lustre: Skipped 2 previous similar messages [ 6416.456251] LustreError: 169748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790104178 with bad export cookie 11864984046420356573 [ 6416.463024] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6416.468137] LustreError: 169748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6416.836014] Lustre: server umount lustre-MDT0001 complete [ 6431.369372] Lustre: server umount lustre-OST0000 complete [ 6452.284873] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 6462.269480] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 6499.169757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6499.386448] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6499.404455] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6499.448193] Lustre: lustre-MDT0000: new disk, initializing [ 6499.492225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6499.507544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6503.968478] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6515.898864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6515.979682] Lustre: 178717:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6516.051691] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6516.058862] Lustre: Skipped 1 previous similar message [ 6516.160507] Lustre: lustre-MDT0001: new disk, initializing [ 6516.279689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6516.324096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6516.335545] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6520.892937] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6531.573586] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6538.006444] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6546.129538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6546.427112] Lustre: lustre-OST0000: new disk, initializing [ 6546.431632] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6546.444653] Lustre: 180687:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6553.546662] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6554.846348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6554.861128] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6554.990366] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6563.720535] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6569.170446] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6569.444131] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6574.544272] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6574.834302] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6580.707108] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6580.708334] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6580.724995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6580.726367] LustreError: Skipped 1 previous similar message [ 6580.733103] Lustre: Skipped 2 previous similar messages [ 6583.142957] Lustre: server umount lustre-OST0000 complete [ 6587.873056] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6587.885730] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6587.897677] Lustre: Skipped 1 previous similar message [ 6587.903681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6593.999847] Lustre: server umount lustre-MDT0000 complete [ 6596.072450] LustreError: 178729:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6596.114302] LustreError: 178729:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [ 6598.694373] LustreError: 178708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790104360 with bad export cookie 11864984046420373597 [ 6598.700200] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6598.712088] LustreError: 178708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6599.126249] Lustre: server umount lustre-MDT0001 complete [ 6609.333721] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 15:12:49 (1790104369) [ 6616.179650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6616.619106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6616.626752] Lustre: Skipped 1 previous similar message [ 6620.783576] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6629.725859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6635.752114] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6644.622197] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6650.749159] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6657.404724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6663.892733] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6671.974348] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6685.496448] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 15:14:05 (1790104445) [ 6690.786964] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6690.802351] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6690.814944] Lustre: Skipped 1 previous similar message [ 6690.824593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6690.827669] Lustre: Skipped 2 previous similar messages [ 6695.544348] Lustre: server umount lustre-OST0000 complete [ 6711.270656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6711.275178] Lustre: Skipped 6 previous similar messages [ 6713.311157] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6713.536414] Lustre: server umount lustre-MDT0000 complete [ 6717.406272] LustreError: 183082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790104479 with bad export cookie 11864984046420374479 [ 6717.417217] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6717.428133] LustreError: 183082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6717.796816] Lustre: server umount lustre-MDT0001 complete [ 6729.232647] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 6732.357423] Key type lgssc unregistered [ 6732.601869] LNet: 186831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6732.613550] LNetError: 186831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6732.632651] LNet: Removed LNI 192.168.203.151@tcp [ 6733.603141] Key type .llcrypt unregistered [ 6733.608815] Key type ._llcrypt unregistered [ 6756.232259] Key type ._llcrypt registered [ 6756.242902] Key type .llcrypt registered [ 6756.388588] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 6757.478051] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6757.580165] alg: No test for adler32 (adler32-zlib) [ 6758.873682] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 6759.270155] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 6760.955116] Key type lgssc registered [ 6762.298337] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6770.729135] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6770.760242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6786.271463] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6791.393678] LustreError: 188091:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.151@tcp: failed processing log, type 4: rc = -110 [ 6796.511535] LustreError: 188091:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6796.526517] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6796.535455] LustreError: MGC192.168.203.151@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6796.554991] LustreError: 188091:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6796.571809] LustreError: 188091:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -22 [ 6796.576623] LustreError: 188091:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 6796.656272] Lustre: server umount lustre-OST0000 complete [ 6796.661835] LustreError: 188091:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6807.467683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6809.015337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6813.807910] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6819.303995] LustreError: 188591:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6819.345087] LustreError: 188591:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6822.993225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6823.374842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6829.150136] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6837.057361] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6843.025770] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6849.787929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6850.076979] LustreError: 190409:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6850.090426] Lustre: 190409:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.151@tcp: invalid local config log lustre-OST0000: rc = -22 [ 6850.236418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6856.614821] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6864.084300] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6867.428121] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6867.445342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6867.449648] Lustre: Skipped 1 previous similar message [ 6867.460958] Lustre: Skipped 2 previous similar messages [ 6872.563413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6872.720681] Lustre: server umount lustre-MDT0000 complete [ 6876.298380] LustreError: 188570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790104637 with bad export cookie 10573691638646793387 [ 6876.304079] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6876.305644] LustreError: 188570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6877.667279] LustreError: 188587:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6877.672402] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6877.717924] Lustre: Skipped 1 previous similar message [ 6877.725927] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6877.731477] LustreError: 189319:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000689436c2 x1877060637193216/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6877.732899] Lustre: Skipped 2 previous similar messages [ 6877.903112] Lustre: server umount lustre-MDT0001 complete [ 6882.231824] Lustre: server umount lustre-OST0000 complete [ 6892.526614] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 6904.757532] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 6944.954364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6945.174925] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6945.221684] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6945.342333] Lustre: lustre-MDT0000: new disk, initializing [ 6945.416573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6945.459832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6950.064311] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6961.377873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6961.492354] Lustre: 193933:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6961.563559] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6961.573541] Lustre: Skipped 1 previous similar message [ 6961.670900] Lustre: lustre-MDT0001: new disk, initializing [ 6961.769334] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6961.832786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6961.854972] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6966.861156] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6976.547834] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6982.363830] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6989.205249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6989.445061] Lustre: lustre-OST0000: new disk, initializing [ 6989.449575] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6989.459989] Lustre: 195900:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6989.568806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6995.000620] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6996.516904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6996.525131] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6996.601759] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7003.559087] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7008.962487] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7009.250711] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7014.043713] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7014.260274] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7016.927960] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7016.938550] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7016.963377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7016.978082] Lustre: Skipped 1 previous similar message [ 7017.011322] Lustre: Skipped 1 previous similar message [ 7022.050304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7022.068955] Lustre: Skipped 1 previous similar message [ 7022.824738] Lustre: server umount lustre-OST0000 complete [ 7027.171903] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7027.179159] Lustre: Skipped 1 previous similar message [ 7032.291148] LustreError: 197239:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7032.313575] Lustre: server umount lustre-MDT0000 complete [ 7035.247706] LustreError: 193924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790104796 with bad export cookie 10573691638646794108 [ 7035.255835] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7035.269556] LustreError: 193924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7035.590962] Lustre: server umount lustre-MDT0001 complete [ 7043.473593] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 15:20:03 (1790104803) [ 7059.012467] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 7061.300679] Key type lgssc unregistered [ 7061.541262] LNet: 198845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7061.546916] LNetError: 198845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7061.567126] LNet: Removed LNI 192.168.203.151@tcp [ 7062.281910] Key type .llcrypt unregistered [ 7062.284442] Key type ._llcrypt unregistered [ 7078.069897] Key type ._llcrypt registered [ 7078.072561] Key type .llcrypt registered [ 7078.167816] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 7079.176203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7079.207217] alg: No test for adler32 (adler32-zlib) [ 7080.246763] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 7080.506764] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 7082.151213] Key type lgssc registered [ 7083.289292] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7090.164629] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7090.186443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7091.541314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7095.522072] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7101.922059] LustreError: 200080:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7101.949854] LustreError: 200080:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7103.228606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7103.578409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7107.090169] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7113.703165] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7118.948992] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7126.000593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7126.124348] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7126.126765] LustreError: 201942:0:(llog.c:1696:llog_backup()) MGC192.168.203.151@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7126.132647] Lustre: 201942:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.151@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7126.215210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7133.081890] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7139.372729] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7141.857513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7141.860291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7141.870502] Lustre: Skipped 2 previous similar messages [ 7141.882025] Lustre: Skipped 2 previous similar messages [ 7146.985420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7146.998014] Lustre: Skipped 2 previous similar messages [ 7147.336155] Lustre: server umount lustre-MDT0000 complete [ 7150.393175] LustreError: 200061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790104912 with bad export cookie 9263902996887494997 [ 7150.400112] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7150.404024] LustreError: 200061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7150.692355] Lustre: server umount lustre-MDT0001 complete [ 7164.635113] Lustre: server umount lustre-OST0000 complete [ 7173.794937] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 7184.555156] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 7219.180431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7219.444694] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7219.477980] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7219.580842] Lustre: lustre-MDT0000: new disk, initializing [ 7219.663968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7219.679648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7224.305286] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7234.815152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7234.993366] Lustre: 205467:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7235.059987] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7235.068638] Lustre: Skipped 1 previous similar message [ 7235.083304] LustreError: 206251:0:(llog.c:1696:llog_backup()) MGC192.168.203.151@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7235.096197] LustreError: 206251:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7235.107250] Lustre: 206251:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.151@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7235.128402] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7235.141844] Lustre: Skipped 1 previous similar message [ 7235.207236] Lustre: lustre-MDT0001: new disk, initializing [ 7235.298452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7235.342300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7235.348989] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7239.534560] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7248.625990] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7254.084469] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7260.771505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7260.979407] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7260.984261] Lustre: Skipped 1 previous similar message [ 7260.988169] LustreError: 207412:0:(llog.c:1696:llog_backup()) MGC192.168.203.151@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7260.998426] LustreError: 207412:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7261.007462] Lustre: 207412:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.151@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7261.013648] Lustre: 207412:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7261.037971] Lustre: lustre-OST0000: new disk, initializing [ 7261.047203] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7261.054741] Lustre: 207438:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7261.128769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7266.799698] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7268.904516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7268.921178] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7269.014084] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7275.595403] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7280.131738] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7280.321648] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7284.400740] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7284.593353] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7289.327961] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7289.347233] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7289.357075] Lustre: Skipped 1 previous similar message [ 7292.446833] Lustre: server umount lustre-OST0000 complete [ 7296.992476] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7297.000077] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7297.016081] Lustre: Skipped 1 previous similar message [ 7297.023763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7299.552246] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7302.559184] Lustre: server umount lustre-MDT0000 complete [ 7304.676146] LustreError: 208109:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7304.698316] LustreError: 208109:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7305.591985] LustreError: 205460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790105067 with bad export cookie 9263902996887495718 [ 7305.599831] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7305.602234] LustreError: 205460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7305.816295] Lustre: server umount lustre-MDT0001 complete [ 7314.547994] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 15:24:34 (1790105074) [ 7330.584474] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 7338.052923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7338.432068] LustreError: 210566:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7338.479106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7341.561032] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7343.585546] LustreError: 210567:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7348.707737] LustreError: 210566:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7349.244694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7353.050202] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7355.450860] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7360.997982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7366.019580] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7373.642845] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7374.006769] Lustre: lustre-OST0001: new disk, initializing [ 7374.020991] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7374.043614] Lustre: 212875:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7374.212236] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7374.222603] Lustre: Skipped 2 previous similar messages [ 7382.060344] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7382.186301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7382.211090] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7382.326686] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7393.293581] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7400.251338] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7402.812408] Lustre: DEBUG MARKER: ADD OST3 [ 7407.540375] loop: module loaded [ 7407.573293] loop0: detected capacity change from 0 to 204800000 [ 7407.727615] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7407.734689] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7407.741509] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7412.818533] loop0: detected capacity change from 0 to 204800000 [ 7414.196308] Lustre: DEBUG MARKER: STOP OST3 [ 7417.800078] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7417.942291] Lustre: lustre-OST0003: new disk, initializing [ 7417.946101] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7417.958097] Lustre: 214818:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7418.932882] Lustre: Failing over lustre-OST0003 [ 7419.029209] Lustre: server umount lustre-OST0003 complete [ 7419.492271] LustreError: 213238:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7423.264677] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7432.123603] Lustre: DEBUG MARKER: START OST3 again [ 7438.162604] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7438.433335] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7438.443882] Lustre: Skipped 1 previous similar message [ 7443.410218] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7443.470279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7443.479396] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7443.509192] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 7463.758381] Lustre: Failing over lustre-OST0003 [ 7463.837637] Lustre: server umount lustre-OST0003 complete [ 7463.912506] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7463.912644] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7463.938929] LustreError: 212052:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7463.955861] LustreError: 212052:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 7482.849980] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7482.858372] LustreError: Skipped 1 previous similar message [ 7482.866816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7482.893170] Lustre: Skipped 1 previous similar message [ 7482.905705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7482.919457] Lustre: Skipped 2 previous similar messages [ 7487.003952] Lustre: server umount lustre-MDT0000 complete [ 7490.722336] LustreError: 213645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790105252 with bad export cookie 9263902996887496600 [ 7490.736452] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7490.738128] LustreError: 213645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7494.631981] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7494.641967] Lustre: Skipped 5 previous similar messages [ 7500.779901] LustreError: 212868:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7500.805871] LustreError: 212868:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 17 previous similar messages [ 7505.375546] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7505.616875] Lustre: server umount lustre-MDT0001 complete [ 7509.386685] Lustre: server umount lustre-OST0000 complete [ 7513.274251] Lustre: server umount lustre-OST0001 complete [ 7520.465828] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 7528.542873] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 7561.481205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7561.702651] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7561.729444] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7561.810361] Lustre: lustre-MDT0000: new disk, initializing [ 7561.905284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7566.377110] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7576.221291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7576.374218] Lustre: 220330:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7576.556945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7576.561691] Lustre: Skipped 1 previous similar message [ 7576.608693] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7581.517876] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7590.523440] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7596.239245] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7602.432849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7602.694159] Lustre: lustre-OST0000: new disk, initializing [ 7602.696893] Lustre: Skipped 1 previous similar message [ 7602.700059] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7602.705608] Lustre: Skipped 2 previous similar messages [ 7602.711834] Lustre: 222299:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7604.679271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7604.687525] Lustre: Skipped 1 previous similar message [ 7604.692108] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7604.812657] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7608.094985] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7615.889546] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7619.689995] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7619.875639] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7623.667817] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7623.891890] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7630.306084] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7630.309867] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7630.312616] LustreError: Skipped 1 previous similar message [ 7630.319511] Lustre: Skipped 5 previous similar messages [ 7630.320858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7630.320863] Lustre: Skipped 4 previous similar messages [ 7631.795811] Lustre: server umount lustre-OST0000 complete [ 7641.350825] Lustre: server umount lustre-MDT0000 complete [ 7644.094672] LustreError: 221124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790105405 with bad export cookie 9263902996887498875 [ 7644.097474] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7644.104296] LustreError: 221124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7644.397829] Lustre: server umount lustre-MDT0001 complete [ 7647.352529] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 15:30:07 (1790105407) [ 7660.078627] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 7668.222928] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 7702.564480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7702.753644] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7702.774204] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7702.832122] Lustre: lustre-MDT0000: new disk, initializing [ 7702.924218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7706.134939] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7715.349883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7715.443079] Lustre: 226908:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7715.720065] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7719.180447] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7726.378808] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7731.281110] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7738.121717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7738.502391] Lustre: 228875:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7739.952372] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7746.285418] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7756.782935] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7760.819870] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7761.029207] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7764.569341] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7764.802494] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7770.592125] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7770.596064] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7770.601493] Lustre: Skipped 3 previous similar messages [ 7770.604918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7770.607887] Lustre: Skipped 5 previous similar messages [ 7772.876414] Lustre: server umount lustre-OST0000 complete [ 7782.679901] Lustre: server umount lustre-MDT0000 complete [ 7785.955691] LustreError: 226914:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7785.998675] LustreError: 226914:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7787.638666] LustreError: 226900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790105549 with bad export cookie 9263902996887499757 [ 7787.641193] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7787.657089] LustreError: 226900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7788.158559] Lustre: server umount lustre-MDT0001 complete [ 7798.540574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7803.644765] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7812.637254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7817.193711] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7824.087841] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7830.003991] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7836.821748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7837.126700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7837.131594] Lustre: Skipped 6 previous similar messages [ 7842.298563] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7849.016482] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7859.680576] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7859.682609] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7859.688706] LustreError: Skipped 3 previous similar messages [ 7859.697221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7859.714357] Lustre: Skipped 4 previous similar messages [ 7859.731337] Lustre: Skipped 5 previous similar messages [ 7864.443811] Lustre: server umount lustre-OST0000 complete [ 7873.856093] Lustre: server umount lustre-MDT0000 complete [ 7877.115868] LustreError: 233968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790105638 with bad export cookie 9263902996887500639 [ 7877.121733] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7877.124477] LustreError: 233968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7877.433125] Lustre: server umount lustre-MDT0001 complete [ 7988.316060] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 7990.552936] Key type lgssc unregistered [ 7990.806090] LNet: 234755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7990.811765] LNetError: 234755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7990.834812] LNet: Removed LNI 192.168.203.151@tcp [ 7991.477148] Key type .llcrypt unregistered [ 7991.479404] Key type ._llcrypt unregistered [ 8015.054128] Key type ._llcrypt registered [ 8015.058319] Key type .llcrypt registered [ 8015.188872] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 8029.420357] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 8030.243045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8030.372880] alg: No test for adler32 (adler32-zlib) [ 8031.367368] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 8031.654022] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 8033.327468] Key type lgssc registered [ 8034.526278] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8069.664291] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8069.712042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8070.903971] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8070.927884] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8071.019361] Lustre: lustre-MDT0000: new disk, initializing [ 8071.109875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8071.128325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8075.723870] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8085.885903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8086.009147] Lustre: 238123:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8086.054792] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8086.058263] Lustre: Skipped 1 previous similar message [ 8086.116439] Lustre: lustre-MDT0001: new disk, initializing [ 8086.204319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8086.241305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8086.251574] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8090.802495] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8099.990321] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8105.483720] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8112.443878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8112.838529] Lustre: lustre-OST0000: new disk, initializing [ 8112.847814] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8112.867313] Lustre: 240092:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8113.017570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8118.565445] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8120.875047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8120.885396] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8120.987058] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8126.966325] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8133.078300] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8133.479845] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8138.391785] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8138.612569] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8141.280710] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8141.292403] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8141.308575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8141.805418] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8141.819728] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8141.840929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8146.834953] Lustre: server umount lustre-OST0000 complete [ 8152.033088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8152.037616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8152.045505] Lustre: Skipped 1 previous similar message [ 8156.792234] Lustre: server umount lustre-MDT0000 complete [ 8160.455190] LustreError: 238117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790105922 with bad export cookie 7012868083867429862 [ 8160.455956] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8160.462490] LustreError: 238117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8160.903805] Lustre: server umount lustre-MDT0001 complete [ 8179.600409] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 8192.002409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8192.332301] LustreError: 243025:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8192.354714] LustreError: 243025:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8192.400377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8197.427114] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8197.605297] LustreError: 243026:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8202.732382] LustreError: 243025:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8205.808178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8206.142677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8211.174498] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8214.580360] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8221.079621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8221.406560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8228.275144] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8236.906967] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8237.125453] Lustre: lustre-OST0001: new disk, initializing [ 8237.128531] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8237.134876] Lustre: 245332:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8243.928440] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8246.912331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8246.926413] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8247.047821] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8255.876451] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8265.259842] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8280.510785] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 15:40:39 (1790106039) [ 8283.105899] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8283.108757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8283.113870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8283.133081] Lustre: Skipped 3 previous similar messages [ 8283.147684] Lustre: Skipped 4 previous similar messages [ 8288.230430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8288.242198] Lustre: Skipped 3 previous similar messages [ 8288.664529] Lustre: server umount lustre-MDT0000 complete [ 8292.397369] LustreError: 246100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790106054 with bad export cookie 7012868083867430751 [ 8292.400446] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8292.410990] LustreError: 246100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8292.697654] Lustre: server umount lustre-MDT0001 complete [ 8306.721164] Lustre: server umount lustre-OST0000 complete [ 8309.727150] Lustre: 236338:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790106055/real 1790106055] req@ffff8a95c2988000 x1877061972112896/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790106071 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8309.768401] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8310.520922] Lustre: server umount lustre-OST0001 complete [ 8317.784482] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 8327.782950] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 8364.610362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8364.953655] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8365.000452] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8365.098206] Lustre: lustre-MDT0000: new disk, initializing [ 8365.191483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8365.196191] Lustre: Skipped 1 previous similar message [ 8365.222772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8369.588396] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8381.131685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8381.285854] Lustre: 249280:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8381.383158] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8381.389114] Lustre: Skipped 1 previous similar message [ 8381.475368] Lustre: lustre-MDT0001: new disk, initializing [ 8381.594176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8381.603676] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8386.336632] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8396.368755] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8402.166555] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8409.161572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8409.472388] Lustre: lustre-OST0000: new disk, initializing [ 8409.476579] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8409.482459] Lustre: 251250:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8409.589573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8409.596213] Lustre: Skipped 1 previous similar message [ 8411.343636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8411.353695] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8411.450541] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8415.457589] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8424.067913] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8428.479452] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8428.651855] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8432.503818] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8432.704412] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8437.216094] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8437.221801] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8437.223747] LustreError: Skipped 1 previous similar message [ 8437.225198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8437.233064] Lustre: Skipped 1 previous similar message [ 8440.854867] Lustre: server umount lustre-OST0000 complete [ 8450.633676] Lustre: server umount lustre-MDT0000 complete [ 8452.576658] LustreError: 250165:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8452.601380] LustreError: 250165:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8454.174809] LustreError: 250084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790106215 with bad export cookie 7012868083867432886 [ 8454.183474] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8454.186296] LustreError: 250084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8454.410887] Lustre: server umount lustre-MDT0001 complete [ 8464.121702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8464.678403] LustreError: 253464:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8470.471778] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8475.108028] LustreError: 253465:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8475.131830] LustreError: 253465:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8483.591329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8484.337712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8484.342188] Lustre: Skipped 1 previous similar message [ 8490.670530] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8498.482385] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8504.052819] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8510.311846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8517.030185] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8524.814530] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8668.645763] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8668.653935] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8668.664920] Lustre: Skipped 3 previous similar messages [ 8668.666440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8668.682961] Lustre: Skipped 4 previous similar messages [ 8673.925540] Lustre: server umount lustre-OST0000 complete [ 8678.881564] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8683.327522] Lustre: server umount lustre-MDT0000 complete [ 8684.002223] LustreError: 253464:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8684.014846] LustreError: 253464:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8686.223415] LustreError: 253447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790106447 with bad export cookie 7012868083867433761 [ 8686.232912] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8686.237329] LustreError: 253447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8686.611374] Lustre: server umount lustre-MDT0001 complete [ 8790.729989] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 8792.622787] Key type lgssc unregistered [ 8792.798575] LNet: 257079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8792.802959] LNetError: 257079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8792.821127] LNet: Removed LNI 192.168.203.151@tcp [ 8793.448247] Key type .llcrypt unregistered [ 8793.453421] Key type ._llcrypt unregistered [ 8812.921514] Key type ._llcrypt registered [ 8812.923486] Key type .llcrypt registered [ 8813.002365] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 8825.064556] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 8825.789108] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8825.881575] alg: No test for adler32 (adler32-zlib) [ 8826.843720] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 8827.014824] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 8828.663257] Key type lgssc registered [ 8829.537910] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8865.752912] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 8873.187311] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8873.200247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8874.373540] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8874.398215] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8874.473726] Lustre: lustre-MDT0000: new disk, initializing [ 8874.537553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8874.556338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8877.534788] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8884.823357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8884.902575] Lustre: 261140:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8884.947249] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8884.950487] Lustre: Skipped 1 previous similar message [ 8885.002773] Lustre: lustre-MDT0001: new disk, initializing [ 8885.077306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8885.106551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8885.111717] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8888.169073] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8891.926889] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8897.197628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8897.394366] Lustre: lustre-OST0000: new disk, initializing [ 8897.398684] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8897.406641] Lustre: 262769:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8897.553332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8897.611865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8897.626825] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8897.775843] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8902.511992] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8911.205600] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8911.361399] Lustre: lustre-OST0001: new disk, initializing [ 8911.370669] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8911.383344] Lustre: 263644:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8911.477143] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8916.662854] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8917.048743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8917.058788] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8917.134195] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8925.883354] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8930.825263] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8941.540362] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8941.547552] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8941.564203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8943.078772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8943.081915] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8943.093513] Lustre: Skipped 1 previous similar message [ 8943.114623] Lustre: Skipped 1 previous similar message [ 8946.774334] Lustre: server umount lustre-MDT0000 complete [ 8950.687939] LustreError: 261133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790106712 with bad export cookie 18402751671775999852 [ 8950.690424] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8950.707306] LustreError: 261133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8951.044506] Lustre: server umount lustre-MDT0001 complete [ 8965.213141] Lustre: server umount lustre-OST0000 complete [ 8979.456528] Lustre: server umount lustre-OST0001 complete [ 8991.331856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8998.825326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9004.934774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9011.927051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9027.549997] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 9036.171143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9036.202588] Lustre: lustre-MDT0000: reset Object Index mappings [ 9036.655065] LustreError: 268654:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9036.678785] LustreError: 268654:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 9036.743764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9040.142854] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9041.889048] LustreError: 268655:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9047.016231] LustreError: 268654:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9047.784626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9047.797682] Lustre: lustre-MDT0001: reset Object Index mappings [ 9048.040326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9050.942313] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9052.990640] Lustre: 269796:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9057.459986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9058.726055] LustreError: 270149:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9061.813817] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9061.931383] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9066.978619] LustreError: 270150:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9066.994649] LustreError: 270150:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9067.850716] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9068.027637] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9068.032859] Lustre: Skipped 1 previous similar message [ 9072.355240] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9073.157812] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9078.465249] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9081.714934] Lustre: 271665:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9088.482159] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9088.493350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9088.493983] Lustre: Skipped 4 previous similar messages [ 9088.508249] Lustre: Skipped 5 previous similar messages [ 9091.607815] Lustre: server umount lustre-MDT0000 complete [ 9093.605809] LustreError: 268655:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9093.624952] LustreError: 268655:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9094.062152] LustreError: 268634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790106855 with bad export cookie 18402751671776006642 [ 9094.072045] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9094.079839] LustreError: 268634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9094.310986] Lustre: server umount lustre-MDT0001 complete [ 9106.991051] Lustre: server umount lustre-OST0000 complete [ 9119.887590] Lustre: server umount lustre-OST0001 complete [ 9130.491318] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 15:54:51 (1790106891) [ 9141.337891] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 9148.922275] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 9176.609523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9176.842613] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9176.869148] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9176.939978] Lustre: lustre-MDT0000: new disk, initializing [ 9177.056584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9177.082292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9180.509982] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9189.145553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9189.266360] Lustre: 275915:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9189.326605] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9189.331641] Lustre: Skipped 1 previous similar message [ 9189.400618] Lustre: lustre-MDT0001: new disk, initializing [ 9189.500617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9189.514457] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9192.493181] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9199.730262] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9204.485989] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9209.528330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9209.734915] Lustre: lustre-OST0000: new disk, initializing [ 9209.737888] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9209.742610] Lustre: 277881:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9211.477858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9211.483057] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9211.578559] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9214.928921] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9222.914498] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9226.440616] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9226.586804] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9229.900483] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9230.136395] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9232.355158] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9232.361681] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9232.366659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9232.379529] LustreError: Skipped 1 previous similar message [ 9237.480172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9237.486591] Lustre: Skipped 1 previous similar message [ 9237.881432] Lustre: server umount lustre-OST0000 complete [ 9242.593759] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9242.615814] Lustre: Skipped 1 previous similar message [ 9247.712575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9247.726108] Lustre: Skipped 4 previous similar messages [ 9255.393706] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9255.608914] Lustre: server umount lustre-MDT0000 complete [ 9257.958839] LustreError: 275926:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9257.979647] LustreError: 275926:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9259.230215] LustreError: 276709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790107020 with bad export cookie 18402751671776008735 [ 9259.230869] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9259.242744] LustreError: 276709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9259.440910] Lustre: server umount lustre-MDT0001 complete [ 9274.666530] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 9283.646163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9284.092723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9284.096704] Lustre: Skipped 2 previous similar messages [ 9287.846670] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9294.308575] LustreError: 280813:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9294.316310] LustreError: 280813:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9294.698228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9298.645868] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9301.102978] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9306.536185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9312.739844] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9321.825719] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9322.102176] Lustre: lustre-OST0001: new disk, initializing [ 9322.111793] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9322.130874] Lustre: 283120:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9324.003978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9324.017579] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9324.138568] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9328.919940] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9338.410669] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9341.607985] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9346.195663] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9346.273721] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9354.724905] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9354.727391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9354.743349] Lustre: Skipped 3 previous similar messages [ 9354.753865] Lustre: Skipped 5 previous similar messages [ 9360.569524] Lustre: server umount lustre-MDT0000 complete [ 9363.843154] LustreError: 280793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790107125 with bad export cookie 18402751671776009610 [ 9363.852335] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9363.859671] LustreError: 280793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9364.191941] Lustre: server umount lustre-MDT0001 complete [ 9377.741768] Lustre: server umount lustre-OST0000 complete [ 9380.831205] Lustre: 258639:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790107126/real 1790107126] req@ffff8a96feb2ca80 x1877062805841024/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790107142 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9380.854776] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9380.865380] Lustre: Skipped 1 previous similar message [ 9381.090577] Lustre: server umount lustre-OST0001 complete [ 9394.595526] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 9402.196435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9402.497186] LustreError: 286463:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9406.384932] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9412.393754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9412.662027] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9412.666144] Lustre: Skipped 4 previous similar messages [ 9416.140623] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9418.233819] Lustre: 287604:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9423.362465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9428.129447] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9428.783456] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9435.078420] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9438.340646] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9439.977759] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9445.916272] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9448.602573] Lustre: 289480:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9452.287202] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9452.396385] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9462.072990] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 16:00:22 (1790107222) [ 9479.136239] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9479.143667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9479.156168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9479.159139] Lustre: Skipped 4 previous similar messages [ 9482.055876] Lustre: server umount lustre-MDT0000 complete [ 9484.466627] LustreError: 286443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790107246 with bad export cookie 18402751671776016239 [ 9484.471429] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9484.477564] LustreError: 286443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9490.792761] Lustre: server umount lustre-MDT0001 complete [ 9499.650427] Lustre: server umount lustre-OST0000 complete [ 9508.797837] Lustre: server umount lustre-OST0001 complete [ 9513.716884] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 9521.301334] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 9560.475968] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 9569.331792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9569.574810] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9569.594942] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9569.690323] Lustre: lustre-MDT0000: new disk, initializing [ 9569.787976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9573.645850] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9582.954786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9583.077310] Lustre: 294014:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9583.313285] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9586.988340] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9596.539270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9596.830923] Lustre: 295648:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9598.029813] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9601.778795] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9610.354743] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9610.504165] Lustre: 296518:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9611.840405] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9614.978452] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9622.658813] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9625.727582] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9625.730223] Lustre: Skipped 1 previous similar message [ 9638.129343] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 16:03:18 (1790107398) [ 9642.348052] Lustre: server umount lustre-MDT0000 complete [ 9642.468259] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9642.474735] LustreError: 294892:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9642.481643] Lustre: Skipped 8 previous similar messages [ 9642.519626] LustreError: 294892:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 9645.667691] LustreError: 294008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790107407 with bad export cookie 18402751671776023960 [ 9645.668950] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9645.679484] LustreError: 294008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9646.087230] Lustre: server umount lustre-MDT0001 complete [ 9660.046207] Lustre: server umount lustre-OST0000 complete [ 9663.968112] Lustre: 258636:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790107409/real 1790107409] req@ffff8a96feb2c000 x1877062805978240/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790107425 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9667.103707] Lustre: server umount lustre-OST0001 complete [ 9682.781330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9683.576228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9683.585554] Lustre: Skipped 6 previous similar messages [ 9690.835855] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9706.238292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9712.892628] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9721.756631] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9728.070848] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9734.929914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9741.890599] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9749.569964] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9770.949077] loop1: detected capacity change from 0 to 204800000 [ 9771.189169] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9771.197673] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9771.230162] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9776.458833] loop1: detected capacity change from 0 to 204800000 [ 9781.910730] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9782.012747] LustreError: 298754:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs12623: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 9782.026712] LustreError: 298754:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs12623 [ 9782.033914] LustreError: 298754:0:(mgs_handler.c:517:mgs_target_reg()) fs12623-MDT0001: Error checking target index: rc = -13 [ 9782.044890] LustreError: fs12623-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 9782.058162] LustreError: 302485:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [ 9782.062939] LustreError: 302485:0:(tgt_mount.c:2128:server_put_super()) no obd fs12623-MDT0001 [ 9782.078332] LustreError: 302485:0:(tgt_mount.c:116:server_deregister_mount()) fs12623-MDT0001 not registered [ 9782.125576] Lustre: server umount fs12623-MDT0001 complete [ 9782.130057] LustreError: 302485:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 9792.597946] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9792.696452] Lustre: Setting parameter fs12623-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs12623-MDT0001 [ 9792.771536] Lustre: srv-fs12623-MDT0001: No data found on store. Initialize space. [ 9792.779691] Lustre: Skipped 4 previous similar messages [ 9792.930984] Lustre: fs12623-MDT0001: new disk, initializing [ 9792.935405] Lustre: Skipped 3 previous similar messages [ 9793.192113] Lustre: srv-fs12623-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 9793.200042] Lustre: 302952:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs12623-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9793.205457] LustreError: 302952:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs12623-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9798.981962] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9806.991879] Lustre: server umount fs12623-MDT0001 complete [ 9821.761692] loop1: detected capacity change from 0 to 204800000 [ 9828.031373] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9828.471837] Lustre: 304522:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs12623-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9828.483897] LustreError: 304522:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs12623-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9830.893083] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [ 9833.307650] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9838.545593] Lustre: server umount fs12623-MDT0001 complete [ 9850.336970] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9850.347981] LustreError: Skipped 1 previous similar message [ 9850.354613] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9850.367694] Lustre: Skipped 1 previous similar message [ 9850.372189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9850.376839] Lustre: Skipped 7 previous similar messages [ 9854.442824] Lustre: server umount lustre-MDT0000 complete [ 9863.035952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9863.170544] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9868.400597] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9868.787339] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9876.105589] loop1: detected capacity change from 0 to 204800000 [ 9881.757650] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9882.062218] Lustre: 306967:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs12623-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9882.076667] LustreError: 306967:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs12623-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9886.855271] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9891.426159] Lustre: server umount fs12623-MDT0001 complete [ 9899.204248] LustreError: 300594:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9899.219478] LustreError: 300594:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 96 previous similar messages [ 9914.873311] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9927.135462] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9927.491783] Lustre: server umount lustre-OST0000 complete [ 9935.336254] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9937.596259] Lustre: server umount lustre-MDT0000 complete [ 9941.331185] LustreError: 298745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790107703 with bad export cookie 18402751671776026641 [ 9941.339604] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9941.352409] LustreError: 298745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9941.661550] Lustre: server umount lustre-MDT0001 complete [ 9953.566772] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 9956.685981] Key type lgssc unregistered [ 9956.942554] LNet: 309310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9956.948754] LNetError: 309310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9956.968733] LNet: Removed LNI 192.168.203.151@tcp [ 9957.895793] Key type .llcrypt unregistered [ 9957.897331] Key type ._llcrypt unregistered [ 9977.518182] Key type ._llcrypt registered [ 9977.519709] Key type .llcrypt registered [ 9977.615099] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 9978.620677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9978.679801] alg: No test for adler32 (adler32-zlib) [ 9979.708702] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 9979.936685] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 9981.567200] Key type lgssc registered [ 9982.937102] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9991.431342] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9991.449713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9993.026848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9997.770336] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10002.416664] LustreError: 310601:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10002.445336] LustreError: 310601:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10006.270435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10006.712723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10011.298915] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10018.952324] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10025.392667] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10032.633848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10033.090777] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10038.189158] LustreError: 312430:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10039.657257] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10043.367933] LustreError: 312712:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10047.120771] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10048.497495] LustreError: 312432:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10055.411830] LustreError: 312712:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10055.430613] LustreError: 312712:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10065.605311] LustreError: 312432:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10065.616619] LustreError: 312432:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [10069.242460] loop1: detected capacity change from 0 to 204800000 [10069.513709] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10069.546707] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10069.577734] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10075.425880] loop1: detected capacity change from 0 to 204800000 [10081.535597] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10081.769723] Lustre: fs11836-OST0000: new disk, initializing [10081.782578] Lustre: srv-fs11836-OST0000: No data found on store. Initialize space. [10081.801064] Lustre: 314335:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10081.947352] Lustre: fs11836-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10082.286357] LustreError: 312432:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10082.319781] LustreError: 312432:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [10088.964954] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10098.287702] Lustre: server umount fs11836-OST0000 complete [10112.588668] loop1: detected capacity change from 0 to 204800000 [10116.816096] LustreError: 312431:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10116.843548] LustreError: 312431:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 18 previous similar messages [10117.696057] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10117.862311] Lustre: fs11836-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10118.130638] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10124.386240] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10132.614854] loop2: detected capacity change from 0 to 204800000 [10132.861968] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10132.873916] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10132.880742] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10133.485400] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10137.508351] loop2: detected capacity change from 0 to 204800000 [10142.799826] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10142.905172] LustreError: fs11836-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10142.923800] LustreError: fs11836-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10142.930557] LustreError: 317375:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [10142.934504] LustreError: 317375:0:(tgt_mount.c:2128:server_put_super()) no obd fs11836-OST0001 [10142.938390] LustreError: 317375:0:(tgt_mount.c:116:server_deregister_mount()) fs11836-OST0001 not registered [10142.958578] Lustre: server umount fs11836-OST0001 complete [10142.962449] LustreError: 317375:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10153.069684] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10153.357082] Lustre: fs11836-OST0001: new disk, initializing [10153.365279] Lustre: srv-fs11836-OST0001: No data found on store. Initialize space. [10153.374198] Lustre: 317926:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10153.511423] Lustre: fs11836-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10160.218783] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10168.384876] Lustre: server umount fs11836-OST0000 complete [10177.794841] Lustre: server umount fs11836-OST0001 complete [10183.363576] LustreError: 312431:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10183.375851] LustreError: 312431:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 38 previous similar messages [10191.882888] loop1: detected capacity change from 0 to 204800000 [10196.464490] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10196.643439] Lustre: fs11836-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10202.394191] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10209.063675] loop2: detected capacity change from 0 to 204800000 [10214.986066] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10220.939803] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10229.004541] Lustre: server umount fs11836-OST0000 complete [10237.537937] Lustre: server umount fs11836-OST0001 complete [10251.234215] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10251.249127] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10251.254679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10251.280883] Lustre: Skipped 1 previous similar message [10256.174987] Lustre: server umount lustre-OST0000 complete [10261.480125] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10261.500962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10261.508760] Lustre: Skipped 2 previous similar messages [10266.072031] Lustre: server umount lustre-MDT0000 complete [10269.868041] LustreError: 310583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790108031 with bad export cookie 15015139275935776974 [10269.869117] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10269.880711] LustreError: 310583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10270.278959] Lustre: server umount lustre-MDT0001 complete [10281.511724] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [10284.206086] Key type lgssc unregistered [10284.481145] LNet: 323847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10284.493489] LNetError: 323847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10284.515568] LNet: Removed LNI 192.168.203.151@tcp [10285.349183] Key type .llcrypt unregistered [10285.353945] Key type ._llcrypt unregistered [10318.047307] Key type ._llcrypt registered [10318.053662] Key type .llcrypt registered [10318.183464] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [10334.291239] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [10335.546255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10335.588414] alg: No test for adler32 (adler32-zlib) [10336.680141] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [10336.953290] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [10338.599519] Key type lgssc registered [10339.794431] Lustre: Echo OBD driver; http://www.lustre.org/ [10377.359090] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10377.381979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10378.704570] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10378.726033] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10378.779769] Lustre: lustre-MDT0000: new disk, initializing [10378.838787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10378.858904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10384.064329] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10395.182158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10395.320442] Lustre: 327724:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10395.393926] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10395.399772] Lustre: Skipped 1 previous similar message [10395.509355] Lustre: lustre-MDT0001: new disk, initializing [10395.671717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10395.712666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10395.749478] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10400.131602] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10410.481071] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10416.965063] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10425.428978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10425.712678] Lustre: lustre-OST0000: new disk, initializing [10425.716258] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10425.722655] Lustre: 329690:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10425.846145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10433.065149] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10434.116985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10434.136078] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10434.288401] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10443.817451] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10445.800659] Lustre: Mounted lustre-client - version 2.17.58_63_gba3a43f [10446.671541] Lustre: Unmounted lustre-client [10456.047949] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10456.053792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10456.060866] Lustre: Skipped 1 previous similar message [10461.161521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10461.172398] Lustre: Skipped 2 previous similar messages [10466.283945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10466.289499] Lustre: Skipped 1 previous similar message [10470.369501] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10470.535854] Lustre: server umount lustre-OST0000 complete [10476.514730] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10476.539768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10476.555404] Lustre: Skipped 1 previous similar message [10480.640123] Lustre: server umount lustre-MDT0000 complete [10484.057730] LustreError: 327716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790108245 with bad export cookie 17202895370725000923 [10484.061827] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10484.066840] LustreError: 327716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10484.482861] Lustre: server umount lustre-MDT0001 complete [10495.181357] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [10498.201625] Key type lgssc unregistered [10505.951817] LNet: 1 local NIs in recovery (showing 1): 192.168.203.151@tcp [10506.783563] LNet: 331855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10506.791343] LNetError: 331855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10577.446282] LNet: Removed LNI 192.168.203.151@tcp [10578.517662] Key type .llcrypt unregistered [10578.524906] Key type ._llcrypt unregistered [10599.551387] Key type ._llcrypt registered [10599.554440] Key type .llcrypt registered [10599.648408] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [10617.182295] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [10618.262102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10618.316703] alg: No test for adler32 (adler32-zlib) [10619.570447] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [10619.778079] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [10621.447192] Key type lgssc registered [10622.790290] Lustre: Echo OBD driver; http://www.lustre.org/ [10658.517318] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 16:20:18 (1790108418) [10666.860623] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10666.921495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10668.160894] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10668.194826] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10668.267327] Lustre: lustre-MDT0000: new disk, initializing [10668.331776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10668.361388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10673.351919] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10683.912919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10684.008684] Lustre: 335094:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10684.087507] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10684.094710] Lustre: Skipped 1 previous similar message [10684.183267] Lustre: lustre-MDT0001: new disk, initializing [10684.292694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10684.330803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10684.349925] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10688.717230] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10699.058668] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10705.986305] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10714.663243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10715.169602] Lustre: lustre-OST0000: new disk, initializing [10715.184292] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10715.203451] Lustre: 337064:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10715.397490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10723.047805] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10724.913018] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10724.920532] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10725.016323] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10732.999501] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10750.957154] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10750.990924] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10750.992624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10751.012626] Lustre: Skipped 1 previous similar message [10754.361740] Lustre: server umount lustre-OST0000 complete [10761.184109] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10761.193596] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10761.194310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10761.199091] Lustre: Skipped 2 previous similar messages [10764.318254] Lustre: server umount lustre-MDT0000 complete [10768.417932] LustreError: 335891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790108530 with bad export cookie 13343080932170263476 [10768.421077] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10768.432398] LustreError: 335891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10768.785313] Lustre: server umount lustre-MDT0001 complete [10779.419049] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [10782.539717] Key type lgssc unregistered [10782.859454] LNet: 338974:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10782.866048] LNetError: 338974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10782.881056] LNet: Removed LNI 192.168.203.151@tcp [10783.782404] Key type .llcrypt unregistered [10783.784667] Key type ._llcrypt unregistered [10803.561193] Key type ._llcrypt registered [10803.562643] Key type .llcrypt registered [10803.685521] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [10804.610471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10804.702547] alg: No test for adler32 (adler32-zlib) [10805.704640] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [10805.887388] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [10807.527386] Key type lgssc registered [10808.642505] Lustre: Echo OBD driver; http://www.lustre.org/ [10816.497716] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10816.536364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10817.957959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10822.582804] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10828.256112] LustreError: 340264:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10828.272327] LustreError: 340264:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10830.947123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10831.364558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10836.309633] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10844.300381] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10850.496505] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10857.123904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10857.606860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10864.077911] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10871.971388] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10886.319242] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10887.864293] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10892.770646] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10892.782590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10892.797339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10895.067714] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.51@tcp (stopping) [10896.865416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10896.867882] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10896.878628] Lustre: Skipped 1 previous similar message [10896.881614] Lustre: Skipped 1 previous similar message [10898.250291] Lustre: server umount lustre-MDT0000 complete [10900.175100] LustreError: 340259:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10902.004193] LustreError: 340245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790108663 with bad export cookie 7760477875795546067 [10902.011313] LustreError: 341254:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10902.021722] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10902.066609] LustreError: 341254:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10905.279942] LustreError: 340260:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10905.335868] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.51@tcp (stopping) [10907.114215] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10908.372400] Lustre: server umount lustre-MDT0001 complete [10912.052988] Lustre: server umount lustre-OST0000 complete [10921.227470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10927.689820] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10930.882658] Lustre: 344446:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.151@tcp [10933.498993] Lustre: server umount MGS complete [10942.934403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10943.451723] LustreError: 345018:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10943.463712] LustreError: 345018:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10943.518338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10947.829969] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10953.696676] LustreError: 345019:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10953.713221] LustreError: 345019:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10955.714734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10955.990281] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10959.970161] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10967.375517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10967.557749] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10972.808698] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10979.440701] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10981.073256] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [10986.909763] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10989.089385] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10995.172822] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [10995.188358] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10995.211968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10995.220569] Lustre: Skipped 1 previous similar message [10996.199625] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10997.148591] Lustre: server umount lustre-OST0000 complete [11002.336679] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11002.341392] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11002.355104] Lustre: Skipped 1 previous similar message [11006.433915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11006.437433] Lustre: Skipped 4 previous similar messages [11007.846797] Lustre: server umount lustre-MDT0000 complete [11011.553665] LustreError: 345015:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11011.564767] LustreError: 345015:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11011.825487] LustreError: 345000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790108773 with bad export cookie 7760477875795546907 [11011.826568] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11011.834737] LustreError: 345000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11012.088540] Lustre: server umount lustre-MDT0001 complete [11023.177299] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [11025.449110] Key type lgssc unregistered [11025.742155] LNet: 348455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11025.755961] LNetError: 348455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11025.773894] LNet: Removed LNI 192.168.203.151@tcp [11026.785659] Key type .llcrypt unregistered [11026.790752] Key type ._llcrypt unregistered [11041.366367] Key type ._llcrypt registered [11041.370744] Key type .llcrypt registered [11041.500408] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [11053.879155] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [11054.727422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11054.770660] alg: No test for adler32 (adler32-zlib) [11055.889019] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [11056.189804] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [11057.841323] Key type lgssc registered [11058.797802] Lustre: Echo OBD driver; http://www.lustre.org/ [11087.112329] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11087.140596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11088.422787] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11088.457467] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11088.541867] Lustre: lustre-MDT0000: new disk, initializing [11088.615841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11088.634311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11092.028486] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11100.174865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11100.337391] Lustre: 351612:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11100.389807] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11100.393817] Lustre: Skipped 1 previous similar message [11100.452508] Lustre: lustre-MDT0001: new disk, initializing [11100.535711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11100.566771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11100.580237] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11104.033037] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11111.726309] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11116.117109] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11121.771310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11121.995352] Lustre: lustre-OST0000: new disk, initializing [11121.998496] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11122.003739] Lustre: 353578:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11122.068777] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11127.307243] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11128.859950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11128.868820] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11128.942602] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11134.510019] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11137.641323] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11137.804277] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11141.246715] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11141.397628] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11144.164384] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11144.165102] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11144.172916] LustreError: Skipped 1 previous similar message [11144.194624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11144.199436] Lustre: Skipped 1 previous similar message [11148.894020] Lustre: server umount lustre-OST0000 complete [11151.840450] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11151.849844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11151.858630] Lustre: Skipped 1 previous similar message [11151.862676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11154.401767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11154.402969] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11157.726967] Lustre: server umount lustre-MDT0000 complete [11160.751713] LustreError: 351603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790108922 with bad export cookie 17306222738583115902 [11160.752888] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11160.760048] LustreError: 351603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11160.947161] Lustre: server umount lustre-MDT0001 complete [11167.621915] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 16:28:48 (1790108928) [11178.772423] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [11185.054926] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [11209.648386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11209.794613] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11209.816857] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11209.878328] Lustre: lustre-MDT0000: new disk, initializing [11209.951480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11209.968727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11213.133459] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11220.356246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11220.456113] Lustre: 358330:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11220.496836] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11220.500753] Lustre: Skipped 1 previous similar message [11220.546740] Lustre: lustre-MDT0001: new disk, initializing [11220.600923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11220.656321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11220.670354] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11223.895274] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11230.429587] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11234.049803] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11238.830399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11238.997141] Lustre: lustre-OST0000: new disk, initializing [11239.000211] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11239.005727] Lustre: 360299:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11239.076879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11243.448583] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11245.087871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11245.093718] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11245.170743] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11249.562581] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11270.625373] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11270.630149] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11270.630437] LustreError: Skipped 1 previous similar message [11270.640124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11270.644900] Lustre: Skipped 2 previous similar messages [11274.307195] Lustre: server umount lustre-OST0000 complete [11277.280855] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11277.284399] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11277.291747] Lustre: Skipped 1 previous similar message [11277.294709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11282.759979] Lustre: server umount lustre-MDT0000 complete [11285.006313] LustreError: 358322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790109046 with bad export cookie 17306222738583116791 [11285.010668] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11285.014038] LustreError: 358322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11285.193443] Lustre: server umount lustre-MDT0001 complete [11291.261101] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [11292.955335] Key type lgssc unregistered [11293.139714] LNet: 362448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11293.151125] LNetError: 362448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11293.164625] LNet: Removed LNI 192.168.203.151@tcp [11293.630333] Key type .llcrypt unregistered [11293.631950] Key type ._llcrypt unregistered [11347.708649] Key type ._llcrypt registered [11347.710878] Key type .llcrypt registered [11347.776110] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [11347.946355] Key type .llcrypt unregistered [11347.948941] Key type ._llcrypt unregistered [11390.347280] Key type ._llcrypt registered [11390.348937] Key type .llcrypt registered [11390.416955] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [11390.570389] Key type .llcrypt unregistered [11390.572735] Key type ._llcrypt unregistered [11411.256814] Key type ._llcrypt registered [11411.269064] Key type .llcrypt registered [11411.500602] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [11411.925407] Key type .llcrypt unregistered [11411.930537] Key type ._llcrypt unregistered [11450.282533] Key type ._llcrypt registered [11450.293249] Key type .llcrypt registered [11450.546254] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [11451.232621] Key type .llcrypt unregistered [11451.234274] Key type ._llcrypt unregistered [11534.093391] Key type ._llcrypt registered [11534.095484] Key type .llcrypt registered [11534.182415] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [11534.380421] Key type .llcrypt unregistered [11534.382472] Key type ._llcrypt unregistered [11563.915426] Key type ._llcrypt registered [11563.917654] Key type .llcrypt registered [11564.001729] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [11564.249325] Key type .llcrypt unregistered [11564.254190] Key type ._llcrypt unregistered [11594.361995] Key type ._llcrypt registered [11594.364248] Key type .llcrypt registered [11594.487982] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [11594.773573] Key type .llcrypt unregistered [11594.779244] Key type ._llcrypt unregistered [11625.206211] Key type ._llcrypt registered [11625.208844] Key type .llcrypt registered [11625.323690] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [11625.612870] Key type .llcrypt unregistered [11625.618994] Key type ._llcrypt unregistered [11656.416655] Key type ._llcrypt registered [11656.418464] Key type .llcrypt registered [11656.502937] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [11656.852970] Key type .llcrypt unregistered [11656.856465] Key type ._llcrypt unregistered [11689.927863] Key type ._llcrypt registered [11689.929935] Key type .llcrypt registered [11690.041995] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [11690.364357] Key type .llcrypt unregistered [11690.369986] Key type ._llcrypt unregistered [11742.273821] Key type ._llcrypt registered [11742.276892] Key type .llcrypt registered [11742.371439] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [11742.538338] Key type .llcrypt unregistered [11742.540075] Key type ._llcrypt unregistered [11761.591689] Key type ._llcrypt registered [11761.593989] Key type .llcrypt registered [11761.661112] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [11761.841366] Key type .llcrypt unregistered [11761.843353] Key type ._llcrypt unregistered [11780.314870] Key type ._llcrypt registered [11780.316096] Key type .llcrypt registered [11780.371599] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [11780.514376] Key type .llcrypt unregistered [11780.516306] Key type ._llcrypt unregistered [11812.665997] Key type ._llcrypt registered [11812.668471] Key type .llcrypt registered [11812.722316] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [11822.080610] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [11822.450883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11822.461133] alg: No test for adler32 (adler32-zlib) [11823.352970] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [11823.474986] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [11825.079198] Key type lgssc registered [11825.611779] Lustre: Echo OBD driver; http://www.lustre.org/ [11846.136053] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11846.154160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11847.375421] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [11847.403403] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [11847.479390] Lustre: tst250fs-MDT0000: new disk, initializing [11847.550661] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11847.564387] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11849.970946] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11856.521144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11856.602674] Lustre: 382549:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11856.647315] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [11856.650263] Lustre: Skipped 1 previous similar message [11856.698594] Lustre: tst250fs-MDT0001: new disk, initializing [11856.745576] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11856.774791] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11856.784461] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11859.389848] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11862.658202] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11866.497919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11866.647533] Lustre: tst250fs-OST0000: new disk, initializing [11866.650513] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [11866.653648] Lustre: 384180:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11866.698841] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11866.824884] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11866.830463] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11866.898748] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11869.643337] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11875.822885] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11875.932366] Lustre: tst250fs-OST0001: new disk, initializing [11875.937584] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [11875.944917] Lustre: 385051:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11875.999771] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11879.565211] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11883.186759] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [11883.389456] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11883.394809] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11883.463715] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11897.825738] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11897.831980] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11897.841697] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [11898.855523] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11898.855678] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [11898.866680] Lustre: Skipped 1 previous similar message [11898.874164] Lustre: Skipped 3 previous similar messages [11903.800523] Lustre: server umount tst250fs-MDT0000 complete [11906.011964] LustreError: 384179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790109667 with bad export cookie 4844738329938277792 [11906.018141] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11906.213810] Lustre: server umount tst250fs-MDT0001 complete [11919.392976] Lustre: server umount tst250fs-OST0000 complete [11931.833865] Lustre: server umount tst250fs-OST0001 complete [11943.472628] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [11948.985778] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [11967.882507] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 16:42:08 (1790109728) [11968.936217] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [11980.026363] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [11987.202675] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [12009.058652] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 11770 sec ========= 16:42:49 (1790109769) [12010.242348] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 16:42:50 (1790109770) === [12012.792513] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 16:42:53 (1790109773) === [12046.729549] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [12048.259352] Key type lgssc unregistered [12048.450678] LNet: 394182:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12048.456654] LNetError: 394182:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12048.465706] LNet: Removed LNI 192.168.203.151@tcp [12048.926126] Key type .llcrypt unregistered [12048.927963] Key type ._llcrypt unregistered