[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 575751964 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002000] Switched APIC routing to physical x2apic. [ 0.002000] kvm-guest: setup PV IPIs [ 0.002000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.002000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.002020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.003007] pid_max: default: 32768 minimum: 301 [ 0.004118] LSM: Security Framework initializing [ 0.005032] Yama: becoming mindful. [ 0.006023] SELinux: Initializing. [ 0.007065] *** VALIDATE selinux *** [ 0.015358] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.019513] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.020121] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.021077] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.022082] *** VALIDATE tmpfs *** [ 0.024043] *** VALIDATE proc *** [ 0.024912] *** VALIDATE cgroup *** [ 0.025005] *** VALIDATE cgroup2 *** [ 0.026208] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.027125] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.028004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.029024] Spectre V2 : User space: Vulnerable [ 0.030004] Speculative Store Bypass: Vulnerable [ 0.033599] debug: unmapping init [mem 0xffffffff9a659000-0xffffffff9a660fff] [ 0.035262] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.036745] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.037019] ... version: 2 [ 0.038009] ... bit width: 48 [ 0.039008] ... generic registers: 4 [ 0.040008] ... value mask: 0000ffffffffffff [ 0.041008] ... max period: 00007fffffffffff [ 0.042007] ... fixed-purpose events: 3 [ 0.043007] ... event mask: 000000070000000f [ 0.044341] rcu: Hierarchical SRCU implementation. [ 0.046555] smp: Bringing up secondary CPUs ... [ 0.047574] x86: Booting SMP configuration: [ 0.048015] .... node #0, CPUs: #1 #2 #3 [ 0.057144] smp: Brought up 1 node, 4 CPUs [ 0.058981] smpboot: Max logical packages: 1 [ 0.059012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.132376] node 0 deferred pages initialised in 55ms [ 0.134098] devtmpfs: initialized [ 0.136218] x86/mm: Memory block size: 128MB [ 0.138944] gcov: version magic: 0x41383552 [ 0.140072] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.141059] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.142234] pinctrl core: initialized pinctrl subsystem [ 0.143129] [ 0.143432] ************************************************************* [ 0.144008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.145008] ** ** [ 0.146008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.147005] ** ** [ 0.148008] ** This means that this kernel is built to expose internal ** [ 0.149011] ** IOMMU data structures, which may compromise security on ** [ 0.150010] ** your system. ** [ 0.151009] ** ** [ 0.152008] ** If you see this message and you are not debugging the ** [ 0.153009] ** kernel, report this immediately to your vendor! ** [ 0.154013] ** ** [ 0.155006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.156005] ************************************************************* [ 0.157638] NET: Registered protocol family 16 [ 0.158407] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.159035] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.160035] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.170038] cpuidle: using governor menu [ 0.174029] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.176471] PCI: Using configuration type 1 for base access [ 0.179130] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.188302] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.190103] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.192150] cryptd: max_cpu_qlen set to 1000 [ 0.195230] ACPI: Added _OSI(Module Device) [ 0.196017] ACPI: Added _OSI(Processor Device) [ 0.198015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.199011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.204947] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.211850] ACPI: Interpreter enabled [ 0.213103] ACPI: PM: (supports S0 S3 S4 S5) [ 0.214009] ACPI: Using IOAPIC for interrupt routing [ 0.216114] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.219447] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.237826] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.241037] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.243015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.247076] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.252045] acpiphp: Slot [2] registered [ 0.253076] acpiphp: Slot [5] registered [ 0.254074] acpiphp: Slot [6] registered [ 0.255051] acpiphp: Slot [7] registered [ 0.256073] acpiphp: Slot [8] registered [ 0.257098] acpiphp: Slot [9] registered [ 0.259063] acpiphp: Slot [10] registered [ 0.259801] acpiphp: Slot [3] registered [ 0.261070] acpiphp: Slot [4] registered [ 0.263094] acpiphp: Slot [11] registered [ 0.269259] acpiphp: Slot [12] registered [ 0.270102] acpiphp: Slot [13] registered [ 0.272267] acpiphp: Slot [14] registered [ 0.275067] acpiphp: Slot [15] registered [ 0.276906] acpiphp: Slot [16] registered [ 0.278302] acpiphp: Slot [17] registered [ 0.281117] acpiphp: Slot [18] registered [ 0.282057] acpiphp: Slot [19] registered [ 0.283071] acpiphp: Slot [20] registered [ 0.284061] acpiphp: Slot [21] registered [ 0.286083] acpiphp: Slot [22] registered [ 0.287184] acpiphp: Slot [23] registered [ 0.289051] acpiphp: Slot [24] registered [ 0.290268] acpiphp: Slot [25] registered [ 0.292074] acpiphp: Slot [26] registered [ 0.293159] acpiphp: Slot [27] registered [ 0.295067] acpiphp: Slot [28] registered [ 0.296054] acpiphp: Slot [29] registered [ 0.299104] acpiphp: Slot [30] registered [ 0.303148] acpiphp: Slot [31] registered [ 0.304070] PCI host bridge to bus 0000:00 [ 0.306013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.308013] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.312015] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.313014] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.317017] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.320016] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.322181] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.325195] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.329685] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.340016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.344034] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.346012] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.348009] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.350011] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.352728] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.358218] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.362064] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.366233] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.378027] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.394008] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.398040] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.405375] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.411016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.420013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.442015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.455082] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.464017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.473018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.500113] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.523622] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.532033] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.537000] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.554016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.564000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.565000] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.572016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.592031] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.605202] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.613016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.622016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.633018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.653341] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.666021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.678022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.695019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.713378] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.715535] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.722078] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.725293] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.726186] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.747056] iommu: Default domain type: Passthrough [ 0.749564] SCSI subsystem initialized [ 0.750176] ACPI: bus type USB registered [ 0.751000] usbcore: registered new interface driver usbfs [ 0.754075] usbcore: registered new interface driver hub [ 0.756081] usbcore: registered new device driver usb [ 0.758154] pps_core: LinuxPPS API ver. 1 registered [ 0.760011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.763167] PTP clock support registered [ 0.765153] EDAC MC: Ver: 3.0.0 [ 0.766000] PCI: Using ACPI for IRQ routing [ 0.766969] NetLabel: Initializing [ 0.769009] NetLabel: domain hash size = 128 [ 0.770009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.772213] NetLabel: unlabeled traffic allowed by default [ 0.775280] vgaarb: loaded [ 0.776651] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.778010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.785416] clocksource: Switched to clocksource kvm-clock [ 0.907423] VFS: Disk quotas dquot_6.6.0 [ 0.908606] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.911057] *** VALIDATE ramfs *** [ 0.912371] *** VALIDATE hugetlbfs *** [ 0.913930] pnp: PnP ACPI init [ 0.916244] pnp: PnP ACPI: found 6 devices [ 0.933530] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.937654] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.940243] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.942673] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.945433] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.950476] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.955740] NET: Registered protocol family 2 [ 0.958491] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.968693] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.972761] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.979461] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.985278] TCP: Hash tables configured (established 65536 bind 65536) [ 0.988786] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.991438] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.994091] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.997801] NET: Registered protocol family 1 [ 1.003737] RPC: Registered named UNIX socket transport module. [ 1.005578] RPC: Registered udp transport module. [ 1.006971] RPC: Registered tcp transport module. [ 1.009400] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.013558] NET: Registered protocol family 44 [ 1.014676] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.017742] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.019570] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.021746] PCI: CLS 0 bytes, default 64 [ 1.023974] Unpacking initramfs... [ 2.902145] debug: unmapping init [mem 0xffff8c6dfcc54000-0xffff8c6dfffbffff] [ 2.905876] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.908242] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.911649] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.461199] Initialise system trusted keyrings [ 3.462301] Key type blacklist registered [ 3.463576] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.470773] zbud: loaded [ 3.473147] *** VALIDATE nfs *** [ 3.474130] *** VALIDATE nfs4 *** [ 3.475452] pstore: using deflate compression [ 3.478254] Platform Keyring initialized [ 3.609581] NET: Registered protocol family 38 [ 3.611853] Key type asymmetric registered [ 3.614351] Asymmetric key parser 'x509' registered [ 3.616246] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.620806] io scheduler mq-deadline registered [ 3.626844] io scheduler kyber registered [ 3.628531] io scheduler bfq registered [ 3.630341] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.634101] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.637333] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.640968] ACPI: Power Button [PWRF] [ 3.647451] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.658195] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.669583] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.676413] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.695254] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.720381] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.745968] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.749943] Non-volatile memory driver v1.3 [ 3.751409] Linux agpgart interface v0.103 [ 3.777949] virtio_blk virtio1: [vda] 149976 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.780073] vda: detected capacity change from 0 to 76787712 [ 4.521263] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.523981] vdb: detected capacity change from 0 to 1073741824 [ 4.539664] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.542442] vdc: detected capacity change from 0 to 2621440000 [ 4.556231] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.558836] vdd: detected capacity change from 0 to 2621440000 [ 4.573959] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.576881] vde: detected capacity change from 0 to 4294967296 [ 4.591649] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.593980] vdf: detected capacity change from 0 to 4294967296 [ 4.600892] libphy: Fixed MDIO Bus: probed [ 4.607264] usbcore: registered new interface driver usbserial_generic [ 4.609303] usbserial: USB Serial support registered for generic [ 4.611533] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.616261] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.618323] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.620538] mousedev: PS/2 mouse device common for all mice [ 4.623364] rtc_cmos 00:05: RTC can wake from S4 [ 4.626788] rtc_cmos 00:05: registered as rtc0 [ 4.628641] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.643419] intel_pstate: CPU model not supported [ 4.646932] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.657381] hid: raw HID events driver (C) Jiri Kosina [ 4.661104] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.664066] usbcore: registered new interface driver usbhid [ 4.667237] usbhid: USB HID core driver [ 4.668071] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.669881] drop_monitor: Initializing network drop monitor service [ 4.676055] Initializing XFRM netlink socket [ 4.677665] NET: Registered protocol family 10 [ 4.681905] Segment Routing with IPv6 [ 4.683192] NET: Registered protocol family 17 [ 4.684687] mpls_gso: MPLS GSO support [ 4.690528] RAS: Correctable Errors collector initialized. [ 4.692371] AVX version of gcm_enc/dec engaged. [ 4.693597] AES CTR mode by8 optimization enabled [ 4.809243] sched_clock: Marking stable (4809221786, 0)->(6000220269, -1190998483) [ 4.812973] registered taskstats version 1 [ 4.814877] Loading compiled-in X.509 certificates [ 4.816793] zswap: loaded using pool lzo/zbud [ 4.844627] Key type big_key registered [ 4.860404] Key type encrypted registered [ 4.861908] ima: No TPM chip found, activating TPM-bypass! [ 4.863981] ima: Allocated hash algorithm: sha1 [ 4.865518] ima: No architecture policies found [ 4.867175] evm: Initialising EVM extended attributes: [ 4.869439] evm: security.selinux [ 4.870945] evm: security.ima [ 4.872445] evm: security.capability [ 4.874356] evm: HMAC attrs: 0x1 [ 4.877430] rtc_cmos 00:05: setting system clock to 2026-09-14 12:11:51 UTC (1789387911) [ 5.186097] debug: unmapping init [mem 0xffffffff9b603000-0xffffffff9b7fffff] [ 5.235293] debug: unmapping init [mem 0xffffffff9a382000-0xffffffff9a658fff] [ 5.256807] Write protecting the kernel read-only data: 28672k [ 5.260497] debug: unmapping init [mem 0xffffffff98a03000-0xffffffff98bfffff] [ 5.263429] debug: unmapping init [mem 0xffffffff99314000-0xffffffff993fffff] [ 5.330626] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.345657] systemd[1]: Detected virtualization kvm. [ 5.350271] systemd[1]: Detected architecture x86-64. [ 5.353183] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.391699] systemd[1]: No hostname configured. [ 5.393558] systemd[1]: Set hostname to . [ 5.398915] random: systemd: uninitialized urandom read (16 bytes read) [ 5.403178] systemd[1]: Initializing machine ID from random generator. [ 5.540225] random: ln: uninitialized urandom read (6 bytes read) [ 5.844715] random: systemd: uninitialized urandom read (16 bytes read) [ 5.851061] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 5.860461] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 5.870044] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.178309] device-mapper: uevent: version 1.0.3 [ 7.196268] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 9.819136] random: fast init done [ 9.922502] virtio_net virtio0 ens2: renamed from eth0 [ 10.274996] scsi host0: ata_piix [ 10.291127] scsi host1: ata_piix [ 10.294167] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.303633] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 15.396208] random: crng init done [ 15.398129] random: 7 urandom warning(s) missed due to ratelimiting [ 19.394744] dracut-initqueue[589]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 21.234930] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 25.898741] printk: systemd: 26 output lines suppressed due to ratelimiting [ 27.417528] SELinux: Disabled at runtime. [ 27.680661] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 27.730184] systemd[1]: Detected virtualization kvm. [ 27.741514] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 30.191118] systemd[1]: initrd-switch-root.service: Succeeded. [ 30.205577] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 30.212843] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 30.222636] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 30.231208] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 30.298997] systemd[1]: Starting Journal Service... Starting Journal Service... [ 30.326591] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Slices. Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ 31.598636] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 33.170951] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 37.027236] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 37.140909] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 38.719406] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 38.905527] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…only root support (10s / no limit) [** ] A start job is running for Configur…only root support (11s / no limit) [*** ] A start job is running for Configur…only root support (12s / no limit) [ *** ] A start job is running for Configur…only root support (12s / no limit) [ *** ] A start job is running for Configur…only root support (13s / no limit) [ ***] A start job is running for Configur…only root support (13s / no limit) [ **] A start job is running for Configur…only root support (14s / no limit) [ *] A start job is running for Configur…only root support (14s / no limit) [ **] A start job is running for Configur…only root support (15s / no limit)[ 45.707713] Key type dns_resolver registered [ ***] A start job is running for Configur…only root support (16s / no limit)[ 46.535978] NFS: Registering the id_resolver key type [ 46.546649] Key type id_resolver registered [ 46.555432] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (16s / no limit) [ *** ] A start job is running for Configur…only root support (17s / no limit) [*** ] A start job is running for Configur…only root support (17s / no limit) [** ] A start job is running for Configur…only root support (18s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ 53.696074] hrtimer: interrupt took 3760357 ns [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [* ] (5 of 6) A start job is running for…SAPI Proxy Daemon (31s / 1min 56s) [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg112-server login: [ 167.743114] libcfs: loading out-of-tree module taints kernel. [ 167.804284] Key type ._llcrypt registered [ 167.806420] Key type .llcrypt registered [ 167.949049] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 187.019633] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 188.742888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 188.755901] alg: No test for adler32 (adler32-zlib) [ 190.324683] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [ 191.469594] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 193.353068] Key type lgssc registered [ 195.083833] Lustre: Echo OBD driver; http://www.lustre.org/ [ 213.053223] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 252.639279] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 264.754310] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 264.781298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 266.036878] LustreError: 6523:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) lustre: New filesystem registration disabled. Use 'lctl set_param allow_register=unlimited' to allow new filesystem registrations: rc = -13 [ 266.048872] Lustre: First target registration detected: opening a one-time 24-hour registration window. Use 'lctl set_param allow_register=0' to close it early, or 'lctl set_param -P allow_register=0' to lock down permanently. [ 266.086742] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 266.117080] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 266.207950] Lustre: lustre-MDT0000: new disk, initializing [ 266.307237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 266.323193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 270.527633] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 276.036551] LustreError: 7055:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 277.723896] Lustre: New target registration enabled indefinitely [ 287.438803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 287.577682] Lustre: 6522:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 287.609821] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 287.613958] Lustre: Skipped 1 previous similar message [ 287.737826] Lustre: lustre-MDT0001: new disk, initializing [ 287.803719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 287.864552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 287.887813] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 291.643711] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 296.555952] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 305.910429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 306.200285] Lustre: lustre-OST0000: new disk, initializing [ 306.203532] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 306.209056] Lustre: 8702:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 306.301910] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 308.247496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 308.260427] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 308.367529] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 313.183346] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 328.548954] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 328.649679] Lustre: lustre-OST0001: new disk, initializing [ 328.653096] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 328.658223] Lustre: 9775:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 328.721870] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 334.977640] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 335.365151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 335.378180] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 335.438388] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 347.859576] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 356.109229] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 363.965703] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing check_logdir /tmp/testlogs/ [ 370.985890] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing yml_node [ 376.320510] Lustre: DEBUG MARKER: Client: 2.17.58.40 [ 379.071229] Lustre: DEBUG MARKER: MDS: 2.17.58.40 [ 381.475772] Lustre: DEBUG MARKER: OSS: 2.17.58.40 [ 383.136602] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Sep 14 08:18:08 EDT 2026 [ 398.163926] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 399.307591] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 415.715194] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 415.727315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 415.743492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 417.248470] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 417.251062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 417.271761] Lustre: Skipped 3 previous similar messages [ 419.220747] Lustre: server umount lustre-MDT0000 complete [ 422.215327] LustreError: 10653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789388328 with bad export cookie 3316225369474407544 [ 422.216208] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 422.226142] LustreError: 10653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 422.368684] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 422.377240] Lustre: Skipped 2 previous similar messages [ 422.383139] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 422.393618] Lustre: Skipped 1 previous similar message [ 427.487769] LustreError: 6528:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 427.504016] LustreError: 6528:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 427.507325] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 427.522983] Lustre: Skipped 1 previous similar message [ 428.557556] Lustre: server umount lustre-MDT0001 complete [ 432.362386] Lustre: server umount lustre-OST0000 complete [ 435.666862] Lustre: server umount lustre-OST0001 complete [ 441.692139] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 450.715202] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 488.058427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 488.262321] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 488.306746] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 488.417456] Lustre: lustre-MDT0000: new disk, initializing [ 488.561728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 488.593508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 493.745566] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 506.416490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 506.628935] Lustre: 15330:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 506.713894] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 506.722891] Lustre: Skipped 1 previous similar message [ 506.815935] Lustre: lustre-MDT0001: new disk, initializing [ 506.987772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 507.040076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 507.045493] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 511.368568] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 522.399623] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 528.726644] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 535.176755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 535.494696] Lustre: lustre-OST0000: new disk, initializing [ 535.497715] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 535.502752] Lustre: 17346:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 535.579771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 537.119226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 537.132681] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 537.232558] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 542.286307] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 551.298278] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 555.794556] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 555.956031] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 561.091965] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 561.295058] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 563.174705] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 563.177161] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 563.184061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 563.189237] Lustre: Skipped 2 previous similar messages [ 563.199734] Lustre: Skipped 1 previous similar message [ 569.367512] Lustre: server umount lustre-OST0000 complete [ 573.412794] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 573.420364] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 573.423202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 573.423211] Lustre: Skipped 2 previous similar messages [ 573.462919] Lustre: Skipped 1 previous similar message [ 578.922607] Lustre: server umount lustre-MDT0000 complete [ 582.590168] LustreError: 15324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789388489 with bad export cookie 3316225369474409077 [ 582.602653] LustreError: 15324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 582.605855] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 582.910343] Lustre: server umount lustre-MDT0001 complete [ 588.194886] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 08:21:33 (1789388493) [ 607.225936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 607.761366] LustreError: 20221:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 607.784067] LustreError: 20221:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 607.869506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 612.371382] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 613.347992] LustreError: 20222:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 617.445330] LustreError: 20221:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 622.585375] LustreError: 20222:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 623.245574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 628.489579] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 636.061828] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 641.031825] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 643.341026] Lustre: Failing over lustre-MDT0000 [ 643.716191] Lustre: server umount lustre-MDT0000 complete [ 644.064964] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 644.068391] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 644.076436] LustreError: 20998:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 644.094660] Lustre: Skipped 1 previous similar message [ 659.428089] Lustre: 3655:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789388550/real 1789388550] req@ffff8c6d422c5180 x1876309414407168/t0(0) o400->MGC192.168.201.112@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789388566 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 659.444135] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 662.516828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 664.544833] LustreError: 20221:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 664.564378] LustreError: 20221:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 669.665604] LustreError: 3653:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8c6e71473b80 x1876309414409856/t0(0) o250->MGC192.168.201.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 669.953636] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 674.389278] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 675.300690] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 675.305744] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 675.334342] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 683.152576] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 689.632368] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 689.647780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 689.659016] Lustre: Skipped 1 previous similar message [ 689.671394] Lustre: Skipped 4 previous similar messages [ 693.650993] Lustre: server umount lustre-MDT0000 complete [ 697.330829] LustreError: 20203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789388603 with bad export cookie 3316225369474410491 [ 697.343290] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 697.347300] LustreError: 20203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 697.903966] Lustre: server umount lustre-MDT0001 complete [ 707.114512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 707.662532] LustreError: 23609:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 707.677812] LustreError: 23609:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 707.731520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 707.735167] Lustre: Skipped 2 previous similar messages [ 713.093558] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 725.663742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 731.763495] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 739.415645] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 746.198324] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 748.650774] Lustre: Failing over lustre-MDT0000 [ 749.081994] Lustre: server umount lustre-MDT0000 complete [ 750.774399] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 750.803755] Lustre: Skipped 1 previous similar message [ 767.392234] Lustre: 3657:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789388657/real 1789388657] req@ffff8c6e789c0380 x1876309414437504/t0(0) o400->MGC192.168.201.112@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789388673 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 767.439300] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 769.432068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 772.577287] LustreError: 23609:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 772.593636] LustreError: 23609:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 16 previous similar messages [ 777.698115] LustreError: 3653:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8c6e7743b100 x1876309414440704/t0(0) o250->MGC192.168.201.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 778.126605] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 782.274735] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 783.328634] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 783.335364] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 783.343524] Lustre: Skipped 1 previous similar message [ 783.374847] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 791.194469] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 796.016863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 801.307982] Lustre: server umount lustre-MDT0000 complete [ 804.559246] LustreError: 23589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789388711 with bad export cookie 3316225369474411429 [ 804.569496] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 805.034474] Lustre: server umount lustre-MDT0001 complete [ 812.703863] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 08:25:17 (1789388717) [ 825.140741] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 832.887784] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 863.752309] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 873.350153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 873.539573] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 873.557923] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 873.626890] Lustre: lustre-MDT0000: new disk, initializing [ 873.713240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 873.715993] Lustre: Skipped 2 previous similar messages [ 873.730379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 877.577436] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 888.300709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 888.465131] Lustre: 30029:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 888.703350] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 893.227369] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 903.226088] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 909.559377] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 916.763412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 916.999726] Lustre: 32049:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 918.371344] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 918.426283] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 918.552986] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 918.580350] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 924.131379] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 934.106284] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 940.060865] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 956.223380] LustreError: 33020:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 964.577711] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 964.581938] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 964.589340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 964.606268] Lustre: Skipped 3 previous similar messages [ 964.617339] Lustre: Skipped 4 previous similar messages [ 965.995972] Lustre: server umount lustre-OST0000 complete [ 976.247537] Lustre: server umount lustre-MDT0000 complete [ 979.668923] LustreError: 30022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789388886 with bad export cookie 3316225369474411835 [ 979.679978] LustreError: 30022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 979.683818] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 980.102867] Lustre: server umount lustre-MDT0001 complete [ 989.348016] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 992.573371] Key type lgssc unregistered [ 992.855728] LNet: 34174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 992.862567] LNetError: 34174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 992.882340] LNet: Removed LNI 192.168.201.112@tcp [ 993.857350] Key type .llcrypt unregistered [ 993.860073] Key type ._llcrypt unregistered [ 1018.172404] Key type ._llcrypt registered [ 1018.174510] Key type .llcrypt registered [ 1018.278218] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 1019.257869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1019.398273] alg: No test for adler32 (adler32-zlib) [ 1020.480861] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [ 1020.879151] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 1022.519254] Key type lgssc registered [ 1023.482699] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1031.048441] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1031.067350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1032.639798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1037.508993] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1041.100370] LustreError: 36056:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1042.914332] LustreError: 35586:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1042.935443] LustreError: 35586:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1042.949532] Lustre: New target registration enabled indefinitely [ 1048.036186] LustreError: 35587:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1050.098473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1050.722279] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1055.315961] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1063.442183] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1069.242356] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1078.454360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1078.947846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1085.653514] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1093.548788] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1096.373594] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1102.530029] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1115.377601] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 08:30:20 (1789389020) [ 1117.360857] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1119.062226] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1125.259379] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 08:30:30 (1789389030) [ 1133.751837] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 08:30:38 (1789389038) [ 1134.814935] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1142.973035] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1142.981240] Lustre: Skipped 2 previous similar messages [ 1144.969081] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 08:30:49 (1789389049) [ 1148.468173] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1148.473911] Lustre: Skipped 3 previous similar messages [ 1165.119192] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1165.123372] Lustre: Skipped 6 previous similar messages [ 1168.172351] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 08:31:12 (1789389072) [ 1170.830297] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1174.223406] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1182.766656] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 08:31:27 (1789389087) [ 1184.896513] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1184.901629] Lustre: Skipped 3 previous similar messages [ 1197.508036] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 08:31:42 (1789389102) [ 1215.458133] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1215.462887] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1215.464532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1215.486638] Lustre: Skipped 1 previous similar message [ 1220.579812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1220.585505] Lustre: Skipped 2 previous similar messages [ 1221.460270] Lustre: server umount lustre-OST0000 complete [ 1225.696684] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1225.708976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1225.714759] Lustre: Skipped 1 previous similar message [ 1230.817330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1230.827766] Lustre: Skipped 1 previous similar message [ 1231.723421] Lustre: server umount lustre-MDT0000 complete [ 1235.941917] LustreError: 35586:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1235.950467] LustreError: 35586:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1235.972855] LustreError: 37674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789389142 with bad export cookie 14008270620669768295 [ 1235.982576] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1235.982677] LustreError: 37674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1236.291195] Lustre: server umount lustre-MDT0001 complete [ 1247.851213] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 1250.915457] Key type lgssc unregistered [ 1251.192169] LNet: 43839:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1251.208253] LNetError: 43839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1252.269273] LNet: Removed LNI 192.168.201.112@tcp [ 1253.218713] Key type .llcrypt unregistered [ 1253.220729] Key type ._llcrypt unregistered [ 1272.450277] Key type ._llcrypt registered [ 1272.452217] Key type .llcrypt registered [ 1272.547569] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 1286.296798] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 1287.195982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1287.342882] alg: No test for adler32 (adler32-zlib) [ 1288.511598] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [ 1288.795074] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 1290.471826] Key type lgssc registered [ 1291.489927] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1322.179673] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1322.216422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1323.415859] LustreError: 46989:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) lustre: New filesystem registration disabled. Use 'lctl set_param allow_register=unlimited' to allow new filesystem registrations: rc = -13 [ 1323.424410] Lustre: First target registration detected: opening a one-time 24-hour registration window. Use 'lctl set_param allow_register=0' to close it early, or 'lctl set_param -P allow_register=0' to lock down permanently. [ 1323.464665] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1323.520875] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1323.629828] Lustre: lustre-MDT0000: new disk, initializing [ 1323.732299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1323.758726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1327.223446] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1332.138199] LustreError: 47522:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1333.574524] Lustre: New target registration enabled indefinitely [ 1339.114261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1339.243954] Lustre: 46987:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1339.285552] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1339.289177] Lustre: Skipped 1 previous similar message [ 1339.344972] Lustre: lustre-MDT0001: new disk, initializing [ 1339.436287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1339.473025] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1339.483253] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1342.410122] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1349.468290] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1353.571606] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1358.511861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1358.702127] Lustre: lustre-OST0000: new disk, initializing [ 1358.705487] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1358.713564] Lustre: 49198:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1358.786737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1363.692113] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1367.088764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1367.096841] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1367.221049] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1371.238936] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1375.374633] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1375.562411] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1380.101722] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1380.254367] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1382.883907] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1382.889095] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1382.914574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1388.002758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1388.006200] Lustre: Skipped 2 previous similar messages [ 1388.477608] Lustre: server umount lustre-OST0000 complete [ 1393.121637] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1393.131134] Lustre: Skipped 1 previous similar message [ 1393.134646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1393.146606] Lustre: Skipped 1 previous similar message [ 1398.224637] Lustre: server umount lustre-MDT0000 complete [ 1401.549668] LustreError: 49197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789389308 with bad export cookie 13325892265668261038 [ 1401.551514] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1401.564438] LustreError: 49197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1401.994770] Lustre: server umount lustre-MDT0001 complete [ 1410.324603] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 08:35:15 (1789389315) [ 1415.732651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1416.126713] LustreError: 51655:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1416.152985] LustreError: 51655:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1416.193487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1419.553985] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1421.280296] LustreError: 51656:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1426.401714] LustreError: 51655:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1427.589683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1427.874403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1431.391354] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1437.249574] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1441.855887] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1449.248576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1449.693050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1455.108079] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1461.829588] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1466.001571] LustreError: 54230:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1466.736472] Lustre: Modifying parameter general.timeout=1 in log params [ 1467.987461] Lustre: Modifying parameter general.timeout=3 in log params [ 1467.990840] Lustre: Skipped 1 previous similar message [ 1470.884504] Lustre: Modifying parameter general.timeout=7 in log params [ 1470.887739] Lustre: Skipped 3 previous similar messages [ 1475.569331] Lustre: Modifying parameter general.timeout=13 in log params [ 1475.574707] Lustre: Skipped 5 previous similar messages [ 1483.614731] Lustre: Modifying parameter general.timeout=23 in log params [ 1483.618570] Lustre: Skipped 9 previous similar messages [ 1499.614837] Lustre: Modifying parameter general.timeout=43 in log params [ 1499.617303] Lustre: Skipped 19 previous similar messages [ 1531.638473] Lustre: Modifying parameter general.timeout=77 in log params [ 1531.641154] Lustre: Skipped 33 previous similar messages [ 1586.452858] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 08:38:11 (1789389491) [ 1589.020910] Lustre: 61164:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1590.351978] Lustre: 61262:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1593.634996] Lustre: 61478:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1593.644792] Lustre: 61478:0:(mgs_llog.c:1453:mgs_modify_param()) Skipped 1 previous similar message [ 1595.797842] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=5000 in log params [ 1595.802551] Lustre: Skipped 59 previous similar messages [ 1617.137681] Lustre: 63026:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1617.143541] Lustre: 63026:0:(mgs_llog.c:1453:mgs_modify_param()) Skipped 2 previous similar messages [ 1617.157708] LustreError: 63026:0:(mgs_handler.c:1214:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1619.983550] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 08:38:45 (1789389525) [ 1628.651356] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1628.653682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1628.666570] Lustre: Skipped 2 previous similar messages [ 1628.676600] Lustre: Skipped 2 previous similar messages [ 1633.764559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1633.775397] Lustre: Skipped 2 previous similar messages [ 1634.790734] Lustre: server umount lustre-MDT0000 complete [ 1637.783194] LustreError: 51635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789389544 with bad export cookie 13325892265668261920 [ 1637.785858] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1637.790696] LustreError: 51635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1638.011135] Lustre: server umount lustre-MDT0001 complete [ 1650.746497] Lustre: server umount lustre-OST0000 complete [ 1672.083146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1672.328857] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1672.559773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1676.460116] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1684.609744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1684.661487] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1684.673922] Lustre: 65557:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1688.728133] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1694.701481] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1699.414556] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1706.060654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1706.179908] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1706.467871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1706.473203] Lustre: Skipped 1 previous similar message [ 1711.621834] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1717.589924] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1719.086166] Lustre: 68167:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1729.504629] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1729.508070] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1729.526443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1732.872810] Lustre: server umount lustre-OST0000 complete [ 1736.163957] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1736.180639] LustreError: Skipped 1 previous similar message [ 1736.183125] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1736.205825] Lustre: Skipped 1 previous similar message [ 1742.246457] Lustre: server umount lustre-MDT0000 complete [ 1745.208026] LustreError: 65551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789389651 with bad export cookie 13325892265668264804 [ 1745.218099] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1745.226198] LustreError: 65551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1745.624934] Lustre: server umount lustre-MDT0001 complete [ 1755.159708] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 1757.839475] Key type lgssc unregistered [ 1758.190889] LNet: 69534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1758.196442] LNetError: 69534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1758.230947] LNet: Removed LNI 192.168.201.112@tcp [ 1758.905307] Key type .llcrypt unregistered [ 1758.907182] Key type ._llcrypt unregistered [ 1782.130654] Key type ._llcrypt registered [ 1782.134450] Key type .llcrypt registered [ 1782.277867] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 1783.324665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1783.405214] alg: No test for adler32 (adler32-zlib) [ 1784.423601] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [ 1784.684940] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 1786.399243] Key type lgssc registered [ 1787.362500] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1794.849213] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1794.879247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1796.353330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1800.603661] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1805.796137] LustreError: 70944:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1805.809889] LustreError: 70944:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1806.286881] Lustre: New target registration enabled indefinitely [ 1810.926756] LustreError: 70943:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1813.748985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1814.369196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1818.865866] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1826.938558] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1832.363365] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1839.032616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1839.523642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1845.774283] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1852.917700] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1863.122589] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1874.413785] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1874.420979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1874.432784] Lustre: Skipped 2 previous similar messages [ 1874.444615] Lustre: Skipped 2 previous similar messages [ 1876.836598] Lustre: server umount lustre-MDT0000 complete [ 1879.523541] LustreError: 70944:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1879.543126] LustreError: 70944:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1881.172514] LustreError: 70925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789389787 with bad export cookie 5123595058883454397 [ 1881.177096] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1881.180164] LustreError: 70925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1881.437993] Lustre: server umount lustre-MDT0001 complete [ 1894.846582] Lustre: server umount lustre-OST0000 complete [ 1925.250975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1925.768758] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1925.819928] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1926.114257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1932.886593] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1943.556611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1943.695892] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1943.727683] Lustre: 76064:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1944.266263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1948.688301] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1955.665572] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1961.606858] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1967.878470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1968.072206] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1968.424371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1974.513932] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1981.357899] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1982.775874] Lustre: 78692:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1984.404320] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1992.170490] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1992.172579] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1992.197362] LustreError: Skipped 1 previous similar message [ 1992.203971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1992.212646] Lustre: Skipped 1 previous similar message [ 1994.158834] Lustre: server umount lustre-OST0000 complete [ 2000.353920] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2000.372997] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2000.402694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2000.411051] Lustre: Skipped 1 previous similar message [ 2002.402854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2002.410093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2002.432909] Lustre: Skipped 1 previous similar message [ 2004.289794] Lustre: server umount lustre-MDT0000 complete [ 2007.523351] LustreError: 76076:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2007.543732] LustreError: 76076:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2007.853284] LustreError: 76058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789389914 with bad export cookie 5123595058883455237 [ 2007.860899] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2007.863374] LustreError: 76058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2008.250924] Lustre: server umount lustre-MDT0001 complete [ 2021.241569] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 2024.122674] Key type lgssc unregistered [ 2024.396235] LNet: 80055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2024.420930] LNetError: 80055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2024.448218] LNet: Removed LNI 192.168.201.112@tcp [ 2025.214308] Key type .llcrypt unregistered [ 2025.215879] Key type ._llcrypt unregistered [ 2050.640905] Key type ._llcrypt registered [ 2050.644754] Key type .llcrypt registered [ 2050.832334] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 2051.921947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2051.946883] alg: No test for adler32 (adler32-zlib) [ 2053.313850] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [ 2053.620375] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 2055.367253] Key type lgssc registered [ 2056.613378] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2063.731177] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2063.749101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2065.283592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2069.761265] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2074.594184] LustreError: 81465:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2074.611924] LustreError: 81465:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2075.385578] Lustre: New target registration enabled indefinitely [ 2079.712913] LustreError: 81466:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2082.869487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2083.389598] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2087.665596] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2095.326493] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2101.641924] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2108.652339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2109.404688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2115.985122] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2123.287163] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2132.191556] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2133.039246] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2134.669810] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2134.676933] Lustre: Skipped 1 previous similar message [ 2137.330774] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2137.334838] Lustre: Skipped 2 previous similar messages [ 2141.736072] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2141.738766] Lustre: Skipped 4 previous similar messages [ 2150.054332] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2150.068074] Lustre: Skipped 8 previous similar messages [ 2166.055029] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2166.059876] Lustre: Skipped 17 previous similar messages [ 2198.258415] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2198.265702] Lustre: Skipped 35 previous similar messages [ 2225.817852] Lustre: 89811:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2228.237986] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 08:48:53 (1789390133) [ 2229.441264] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2231.009431] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 08:48:56 (1789390136) [ 2238.432114] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2238.434891] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2238.454189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2240.484202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2240.489916] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2240.498931] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2242.574748] Lustre: server umount lustre-OST0000 complete [ 2247.152868] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2247.162667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2247.170438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2247.173612] Lustre: Skipped 1 previous similar message [ 2250.723680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2250.726035] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2251.984509] Lustre: server umount lustre-MDT0000 complete [ 2255.104971] LustreError: 81444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789390161 with bad export cookie 3843595913053293840 [ 2255.125397] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2255.448310] Lustre: server umount lustre-MDT0001 complete [ 2267.739639] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 2270.921207] Key type lgssc unregistered [ 2271.223707] LNet: 91219:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2271.232322] LNetError: 91219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2271.247170] LNet: Removed LNI 192.168.201.112@tcp [ 2272.002861] Key type .llcrypt unregistered [ 2272.005358] Key type ._llcrypt unregistered [ 2277.676525] Key type ._llcrypt registered [ 2277.685806] Key type .llcrypt registered [ 2277.861613] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2282.428094] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules [ 2283.333022] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2283.466822] alg: No test for adler32 (adler32-zlib) [ 2284.514539] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [ 2284.678387] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 2286.351188] Key type lgssc registered [ 2287.751209] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2288.828905] LustreError: 92417:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2305.577714] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 2310.648029] LustreError: 92417:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2313.904674] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2313.917774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2315.323708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2319.619356] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2324.406055] Lustre: New target registration enabled indefinitely [ 2324.451517] LustreError: 93442:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2324.458575] LustreError: 93442:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2329.575408] LustreError: 93443:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2332.071338] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 08:50:36 (1789390236) [ 2334.688723] LustreError: 93442:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2339.301065] LustreError: 93474:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2339.312276] LustreError: 93474:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2339.319904] LustreError: 93474:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 24, retries 0, failed: rc = -5 [ 2339.631899] Lustre: server umount lustre-MDT0000 complete [ 2352.042778] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 2354.663845] Key type lgssc unregistered [ 2354.963600] LNet: 95381:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2354.970706] LNetError: 95381:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2354.985746] LNet: Removed LNI 192.168.201.112@tcp [ 2355.905132] Key type .llcrypt unregistered [ 2355.908501] Key type ._llcrypt unregistered [ 2373.139548] Key type ._llcrypt registered [ 2373.141724] Key type .llcrypt registered [ 2373.221332] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 2374.426034] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2374.441865] alg: No test for adler32 (adler32-zlib) [ 2375.492897] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [ 2375.685733] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 2377.295304] Key type lgssc registered [ 2378.472846] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2385.603139] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2385.614794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2387.003528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2391.058442] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2396.276750] Lustre: New target registration enabled indefinitely [ 2397.169200] LustreError: 96597:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2397.193363] LustreError: 96597:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2402.274480] LustreError: 96598:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2403.126633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2403.894331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2408.268917] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2415.222818] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2421.108238] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2426.656079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2427.226260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2432.591292] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2439.458441] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2479.955539] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 08:53:05 (1789390385) [ 2481.550497] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2483.259676] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 08:53:08 (1789390388) [ 2488.802561] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2488.806648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2488.814124] Lustre: Skipped 1 previous similar message [ 2488.834706] Lustre: Skipped 2 previous similar messages [ 2493.551839] Lustre: server umount lustre-MDT0000 complete [ 2493.922658] LustreError: 96597:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2493.936609] LustreError: 96597:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2496.992785] LustreError: 96577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789390403 with bad export cookie 13672885545924874418 [ 2496.999206] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2497.002556] LustreError: 96577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2497.192076] Lustre: server umount lustre-MDT0001 complete [ 2511.307288] Lustre: server umount lustre-OST0000 complete [ 2522.152325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2522.506370] LustreError: 101125:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2522.649789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2526.618231] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2527.722900] LustreError: 101126:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2535.697409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2536.309333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2540.525991] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2547.582902] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2552.990790] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2566.861121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2567.077558] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2567.089102] LustreError: 101114:0:(mgs_handler.c:613:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2567.115314] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2567.121891] LustreError: 103258:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2567.124477] LustreError: 103258:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2567.143400] LustreError: 103258:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2567.223604] Lustre: server umount lustre-OST0000 complete [ 2567.227439] LustreError: 103258:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2575.674524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2575.799570] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2575.803311] LustreError: 101113:0:(mgs_handler.c:613:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2575.818482] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2575.825641] LustreError: 103724:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2575.831349] LustreError: 103724:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2575.835356] LustreError: 103724:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2575.888563] Lustre: server umount lustre-OST0000 complete [ 2575.890894] LustreError: 103724:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2584.802754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2585.011816] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2585.015595] Lustre: Found index 0 for lustre-OST0000, updating log [ 2585.020553] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2585.082250] Lustre: lustre-OST0000: new disk, initializing [ 2585.085723] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2585.091988] Lustre: 104257:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2585.464249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2592.454852] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2596.046927] Lustre: Failing over lustre-OST0000 [ 2596.446361] Lustre: server umount lustre-OST0000 complete [ 2602.466515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2602.488789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2602.519398] Lustre: Skipped 1 previous similar message [ 2602.536670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2606.050711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2606.063993] Lustre: Skipped 2 previous similar messages [ 2610.342241] Lustre: server umount lustre-MDT0000 complete [ 2611.172438] LustreError: 101122:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2611.185424] LustreError: 101122:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2613.649518] LustreError: 101916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789390520 with bad export cookie 13672885545924877162 [ 2613.659478] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2613.667860] LustreError: 101916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2614.091956] Lustre: server umount lustre-MDT0001 complete [ 2621.703979] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 08:55:26 (1789390526) [ 2635.869707] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 2643.448338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2643.754840] LustreError: 106601:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2643.879394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2647.437966] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2654.483915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2658.359556] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2661.166094] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2667.771474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2668.643619] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2673.638600] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2673.692247] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2673.692390] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2673.710996] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2674.398370] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2683.531694] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2683.631586] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2683.692466] Lustre: lustre-OST0001: new disk, initializing [ 2683.696623] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2683.707269] Lustre: 108970:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2684.060401] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2684.066190] Lustre: Skipped 2 previous similar messages [ 2691.342143] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2692.928481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2692.952535] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2693.194499] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2701.324952] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2707.654564] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2713.583817] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2713.589928] Lustre: Skipped 1 previous similar message [ 2713.593136] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2713.596587] Lustre: Skipped 1 previous similar message [ 2718.508302] Lustre: server umount lustre-MDT0001 complete [ 2718.697101] LustreError: 109618:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2718.730174] LustreError: 109618:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2728.796744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2728.913115] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2728.925576] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2729.038156] Lustre: 106591:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2730.256554] LustreError: 110511:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2730.270110] LustreError: 110511:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2730.283980] LustreError: 106593:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2730.415948] LustreError: 110523:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.112@tcp: cfg command failed: rc = -17 [ 2730.426433] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2730.426433] [ 2734.288229] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2735.102238] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2743.764498] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 08:57:28 (1789390648) [ 2761.858431] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 2779.949844] Lustre: 112584:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2800.609680] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2804.342428] Lustre: 113719:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2877.923939] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2877.934668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2877.942642] Lustre: Skipped 2 previous similar messages [ 2877.950241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2877.962226] Lustre: Skipped 2 previous similar messages [ 2882.384641] Lustre: server umount lustre-MDT0000 complete [ 2883.553527] LustreError: 106598:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2883.575705] LustreError: 106598:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 2885.865311] LustreError: 108969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789390792 with bad export cookie 13672885545924878163 [ 2885.870134] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2885.882940] LustreError: 108969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2886.121581] Lustre: server umount lustre-MDT0001 complete [ 2898.695818] Lustre: server umount lustre-OST0000 complete [ 2912.439242] Lustre: server umount lustre-OST0001 complete [ 2917.724287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2927.172951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2935.354868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2944.307176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2964.591689] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 2975.240221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2975.287834] Lustre: lustre-MDT0000: reset Object Index mappings [ 2975.868918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2975.872084] Lustre: Skipped 1 previous similar message [ 2980.117592] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2988.944382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2988.964452] Lustre: lustre-MDT0001: reset Object Index mappings [ 2994.295701] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2996.497668] Lustre: 118872:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3002.742861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3008.802262] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3013.668576] LustreError: 119226:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3013.680429] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3013.684090] LustreError: 119226:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3016.404160] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3021.193550] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:386 to 0x280000401:418) [ 3021.197994] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:225) [ 3021.286991] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3022.946612] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3030.806647] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3034.719331] Lustre: 120749:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3080.285967] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 09:03:04 (1789390984) [ 3107.298148] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3107.307506] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3107.327716] Lustre: Skipped 3 previous similar messages [ 3107.354637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3107.360533] Lustre: Skipped 4 previous similar messages [ 3113.466808] Lustre: server umount lustre-MDT0000 complete [ 3118.300393] LustreError: 120751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789391024 with bad export cookie 13672885545924925238 [ 3118.311841] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3118.326552] LustreError: 120751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3118.562132] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3118.585140] Lustre: Skipped 3 previous similar messages [ 3123.701977] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3123.711045] Lustre: Skipped 7 previous similar messages [ 3125.420817] Lustre: server umount lustre-MDT0001 complete [ 3144.159180] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3144.330431] Lustre: server umount lustre-OST0000 complete [ 3163.103150] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3163.475504] Lustre: server umount lustre-OST0001 complete [ 3170.969090] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 3181.935977] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 3218.957442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3219.180993] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3219.218082] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3219.316363] Lustre: lustre-MDT0000: new disk, initializing [ 3219.405702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3219.408665] Lustre: Skipped 3 previous similar messages [ 3219.436856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3224.479808] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3233.352995] Lustre: server umount lustre-MDT0000 complete [ 3243.806685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3244.129302] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3244.146177] Lustre: 125067:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3244.156909] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3249.396390] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3256.050647] Lustre: server umount lustre-MDT0000 complete [ 3264.964461] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 09:06:09 (1789391169) [ 3266.961174] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3269.119576] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 09:06:13 (1789391173) [ 3289.382710] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 3300.645508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3306.170969] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3317.310123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3317.461468] Lustre: 127168:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3317.519329] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3317.523730] Lustre: Skipped 1 previous similar message [ 3317.618437] Lustre: lustre-MDT0001: new disk, initializing [ 3317.831713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3317.851088] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3322.864305] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3329.112690] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3336.306201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3336.633555] Lustre: lustre-OST0000: new disk, initializing [ 3336.637051] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3336.645353] Lustre: 128799:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3338.278555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3338.283464] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3338.403234] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3343.176740] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3354.048738] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3354.259413] Lustre: lustre-OST0001: new disk, initializing [ 3354.264316] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3354.270718] Lustre: 129669:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3356.265214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3356.276469] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3356.390245] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3361.353938] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3371.250712] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3375.061631] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3763.719310] Lustre: 136080:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3763.725814] Lustre: 136080:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3763.729343] Lustre: 136080:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3763.741894] Lustre: 136080:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3763.755102] Lustre: 136080:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3763.766491] Lustre: 136080:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4039.587535] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:19:03 (1789391943) [ 4058.082604] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4058.089788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4058.105504] Lustre: Skipped 4 previous similar messages [ 4058.113438] Lustre: Skipped 2 previous similar messages [ 4062.262950] Lustre: server umount lustre-MDT0000 complete [ 4063.205797] LustreError: 127176:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4063.229803] LustreError: 127176:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 4065.827669] LustreError: 127162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789391972 with bad export cookie 13672885545924991374 [ 4065.832091] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4065.838054] LustreError: 127162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4068.333946] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4068.338059] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4068.349460] Lustre: Skipped 1 previous similar message [ 4068.377654] Lustre: Skipped 2 previous similar messages [ 4072.562232] Lustre: server umount lustre-MDT0001 complete [ 4082.801907] Lustre: server umount lustre-OST0000 complete [ 4092.718982] Lustre: server umount lustre-OST0001 complete [ 4099.660680] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 4110.444507] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 4146.772905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4147.084421] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4147.139983] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4147.260350] Lustre: lustre-MDT0000: new disk, initializing [ 4147.423966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4147.432113] Lustre: Skipped 5 previous similar messages [ 4147.451742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4151.793681] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4163.822822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4163.955816] Lustre: 139130:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4164.000901] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4164.004554] Lustre: Skipped 1 previous similar message [ 4164.063411] Lustre: lustre-MDT0001: new disk, initializing [ 4164.222829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4164.239291] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4169.388515] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4181.152364] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4187.249190] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4194.443574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4194.854356] Lustre: lustre-OST0000: new disk, initializing [ 4194.865562] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4194.886845] Lustre: 141145:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4195.003721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4195.016587] Lustre: Skipped 1 previous similar message [ 4196.869577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4196.911076] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4197.122435] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4203.001131] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4214.305836] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4224.839142] Lustre: lustre-MDD0000: changelog on [ 4227.454835] Lustre: lustre-MDD0001: changelog on [ 5923.025350] Lustre: lustre-MDD0001: changelog off [ 5924.952040] Lustre: 139138:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5925.769093] Lustre: lustre-MDD0000: changelog off [ 5930.793790] LustreError: 149248:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 5938.145605] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5938.153989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5938.154181] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5938.167485] Lustre: Skipped 1 previous similar message [ 5938.192518] Lustre: Skipped 3 previous similar messages [ 5941.298834] Lustre: server umount lustre-OST0000 complete [ 5945.824979] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5945.827570] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5945.843942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5948.386771] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5948.389260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5948.428544] Lustre: Skipped 1 previous similar message [ 5953.506496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5953.518961] Lustre: Skipped 1 previous similar message [ 5959.135191] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5959.463682] Lustre: server umount lustre-MDT0000 complete [ 5962.724427] LustreError: 142778:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5962.750043] LustreError: 142778:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 5963.739300] LustreError: 139122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789393870 with bad export cookie 13672885545925812131 [ 5963.752685] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5963.758619] LustreError: 139122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5964.268269] Lustre: server umount lustre-MDT0001 complete [ 5974.233596] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 5976.914601] Key type lgssc unregistered [ 5977.182355] LNet: 150399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5977.190362] LNetError: 150399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5978.227074] LNet: Removed LNI 192.168.201.112@tcp [ 5979.146187] Key type .llcrypt unregistered [ 5979.152931] Key type ._llcrypt unregistered [ 5998.053298] Key type ._llcrypt registered [ 5998.055718] Key type .llcrypt registered [ 5998.176285] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 6014.000375] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 6015.053814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6015.090599] alg: No test for adler32 (adler32-zlib) [ 6016.396411] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [ 6016.607420] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 6018.311235] Key type lgssc registered [ 6019.347771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6056.391362] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6056.421762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6057.679242] LustreError: 153614:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) lustre: New filesystem registration disabled. Use 'lctl set_param allow_register=unlimited' to allow new filesystem registrations: rc = -13 [ 6057.690975] Lustre: First target registration detected: opening a one-time 24-hour registration window. Use 'lctl set_param allow_register=0' to close it early, or 'lctl set_param -P allow_register=0' to lock down permanently. [ 6057.707266] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6057.726469] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6057.781684] Lustre: lustre-MDT0000: new disk, initializing [ 6057.831504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6057.844557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6061.280654] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6066.621622] LustreError: 154146:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 6068.156644] Lustre: New target registration enabled indefinitely [ 6074.633876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6074.759981] Lustre: 153614:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6074.806760] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6074.810462] Lustre: Skipped 1 previous similar message [ 6074.882693] Lustre: lustre-MDT0001: new disk, initializing [ 6074.997648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6075.036067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6075.053346] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6079.484524] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6089.071996] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6094.655854] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6100.951053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6101.224319] Lustre: lustre-OST0000: new disk, initializing [ 6101.226942] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6101.233659] Lustre: 155822:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6101.321866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6107.580842] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6108.267578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6108.278476] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6108.446624] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6115.921865] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6121.837230] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_module obdecho/obdecho [ 6123.263454] LustreError: 156828:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6123.270772] LustreError: 156828:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6124.654897] LustreError: 156924:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6136.289444] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6136.294652] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6136.312500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6139.363093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6139.369605] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6142.675760] Lustre: server umount lustre-MDT0000 complete [ 6147.647992] LustreError: 153605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789394054 with bad export cookie 16979910480396331609 [ 6147.653909] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6147.664495] LustreError: 153605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6148.158833] Lustre: server umount lustre-MDT0001 complete [ 6163.283756] Lustre: server umount lustre-OST0000 complete [ 6186.965809] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 6200.238691] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 6240.377799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6240.554660] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6240.577172] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6240.643551] Lustre: lustre-MDT0000: new disk, initializing [ 6240.723962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6240.740070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6245.408885] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6258.526038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6258.698660] Lustre: 160641:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6258.765037] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6258.768773] Lustre: Skipped 1 previous similar message [ 6258.834900] Lustre: lustre-MDT0001: new disk, initializing [ 6258.932439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6258.977653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6258.990744] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6263.721633] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6273.349777] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6278.409777] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6285.487500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6285.704821] Lustre: lustre-OST0000: new disk, initializing [ 6285.707837] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6285.713877] Lustre: 162655:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6285.778685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6292.869485] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6293.067858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6293.077574] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6293.252460] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6302.095854] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6306.292052] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6306.526954] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6311.332308] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6311.591176] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6313.953641] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6313.958251] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6313.984833] Lustre: Skipped 1 previous similar message [ 6313.996294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6313.996302] Lustre: Skipped 2 previous similar messages [ 6319.076819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6319.087052] Lustre: Skipped 2 previous similar messages [ 6319.963901] Lustre: server umount lustre-OST0000 complete [ 6324.193318] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6324.202775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6324.213238] Lustre: Skipped 1 previous similar message [ 6329.965783] Lustre: server umount lustre-MDT0000 complete [ 6333.736715] LustreError: 160631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789394240 with bad export cookie 16979910480396332568 [ 6333.755052] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6333.760571] LustreError: 160631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6334.145663] Lustre: server umount lustre-MDT0001 complete [ 6337.322519] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 09:57:22 (1789394242) [ 6339.194835] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6341.625283] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 09:57:26 (1789394246) [ 6351.519477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6352.009555] LustreError: 165069:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6352.024038] LustreError: 165069:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 6352.116399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6356.993768] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6357.476455] LustreError: 165070:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6361.570079] LustreError: 165069:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6366.224954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6370.936533] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6378.325944] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6384.648933] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6392.010764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6392.289400] LustreError: 166937:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6392.370709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6392.381411] Lustre: Skipped 1 previous similar message [ 6398.550393] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6405.925657] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6412.774741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6412.786429] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6412.811587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6412.816360] Lustre: Skipped 3 previous similar messages [ 6418.343971] Lustre: server umount lustre-MDT0000 complete [ 6421.696506] LustreError: 165050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789394328 with bad export cookie 16979910480396333443 [ 6421.708093] LustreError: 165050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6421.708601] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6421.923548] LustreError: 165066:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6421.985395] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6422.001725] Lustre: Skipped 2 previous similar messages [ 6432.146699] LustreError: 167436:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6432.149959] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.12@tcp (stopping) [ 6432.170256] LustreError: 167436:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6432.190987] Lustre: Skipped 8 previous similar messages [ 6436.321782] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6436.550254] Lustre: server umount lustre-MDT0001 complete [ 6441.523679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6451.198278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6451.427784] LustreError: 168961:0:(llog.c:1655:llog_backup()) MGC192.168.201.112@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6451.432757] Lustre: 168961:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.112@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6461.777847] LustreError: 168966:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6461.806345] LustreError: 168966:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6497.249147] LustreError: 166948:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6497.267808] LustreError: 166948:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 29 previous similar messages [ 6512.095526] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6512.099045] Lustre: 168961:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.112@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6553.091168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6557.981271] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6558.179854] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6563.240507] LustreError: 168967:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6563.260340] LustreError: 168967:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 41 previous similar messages [ 6566.711995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6567.393043] LustreError: 151831:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006341e2b2 x1876315522329728/t0(0) o250->MGC192.168.201.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6572.625452] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6573.031361] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6581.227122] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6582.762170] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6588.489445] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6590.185550] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6607.549609] Lustre: server umount lustre-MDT0000 complete [ 6608.864370] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6608.871755] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6608.890781] Lustre: Skipped 1 previous similar message [ 6611.774932] LustreError: 167899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789394518 with bad export cookie 16979910480396339904 [ 6611.788104] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6611.790723] LustreError: 167899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6612.109732] Lustre: server umount lustre-MDT0001 complete [ 6626.639333] Lustre: server umount lustre-OST0000 complete [ 6649.899492] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 6661.357391] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 6702.955138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6703.458851] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6703.501102] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6703.604394] Lustre: lustre-MDT0000: new disk, initializing [ 6703.740825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6703.744027] Lustre: Skipped 1 previous similar message [ 6703.789472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6708.516512] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6722.189858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6722.296101] Lustre: 174123:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6722.684426] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6728.546709] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6739.156835] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6745.438463] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6752.768110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6753.094908] Lustre: lustre-OST0000: new disk, initializing [ 6753.101033] Lustre: Skipped 1 previous similar message [ 6753.122976] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6753.138886] Lustre: Skipped 2 previous similar messages [ 6753.151900] Lustre: 176141:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6759.685205] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6761.123722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6761.134345] Lustre: Skipped 1 previous similar message [ 6761.152042] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6761.295323] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6768.932882] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6774.204115] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6774.431909] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6778.406527] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6778.588669] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6781.922203] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6781.923962] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6781.930781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6781.930791] Lustre: Skipped 1 previous similar message [ 6781.935233] LustreError: Skipped 1 previous similar message [ 6782.008173] Lustre: Skipped 2 previous similar messages [ 6786.742056] Lustre: server umount lustre-OST0000 complete [ 6796.972573] Lustre: server umount lustre-MDT0000 complete [ 6797.280610] LustreError: 174134:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6797.334993] LustreError: 174134:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 6803.424102] LustreError: 174115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789394710 with bad export cookie 16979910480396356739 [ 6803.456512] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6803.468996] LustreError: 174115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6804.242063] Lustre: server umount lustre-MDT0001 complete [ 6819.310202] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 10:05:23 (1789394723) [ 6828.822550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6834.703811] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6847.101370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6851.497872] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6858.825449] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6864.910461] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6871.498569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6878.196883] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6885.202919] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6894.906922] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 10:06:39 (1789394799) [ 6897.406336] LustreError: 181326:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 6903.235864] Lustre: server umount lustre-OST0000 complete [ 6908.896595] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6908.899046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6908.903261] Lustre: Skipped 2 previous similar messages [ 6908.908585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6908.910594] Lustre: Skipped 3 previous similar messages [ 6917.160313] Lustre: server umount lustre-MDT0000 complete [ 6920.166617] Lustre: 151832:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789394810/real 1789394810] req@0000000048e087f9 x1876315522427008/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789394826 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6921.073025] LustreError: 181524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789394827 with bad export cookie 16979910480396357621 [ 6921.082070] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6921.089427] LustreError: 181524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6921.617577] Lustre: server umount lustre-MDT0001 complete [ 6932.574939] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 6935.779933] Key type lgssc unregistered [ 6936.075824] LNet: 182476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6936.082794] LNetError: 182476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6936.103925] LNet: Removed LNI 192.168.201.112@tcp [ 6936.933571] Key type .llcrypt unregistered [ 6936.936277] Key type ._llcrypt unregistered [ 6956.691201] Key type ._llcrypt registered [ 6956.695167] Key type .llcrypt registered [ 6956.815542] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 6958.444057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6958.456770] alg: No test for adler32 (adler32-zlib) [ 6959.613374] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [ 6959.875885] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 6961.583228] Key type lgssc registered [ 6962.637567] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6970.241680] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6970.281993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6985.951405] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6991.071971] LustreError: 183718:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.112@tcp: failed processing log, type 4: rc = -110 [ 6996.191515] LustreError: 183718:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6996.205139] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6996.220907] LustreError: MGC192.168.201.112@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6996.248645] LustreError: 183718:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6996.265749] LustreError: 183718:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -22 [ 6996.281583] LustreError: 183718:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 6996.388934] Lustre: server umount lustre-OST0000 complete [ 6996.395417] LustreError: 183718:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7005.953312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7007.348118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7012.483477] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7016.558882] LustreError: 184688:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 7016.937969] LustreError: 184219:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7016.949996] LustreError: 184219:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7018.507227] Lustre: New target registration enabled indefinitely [ 7022.053094] LustreError: 184218:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7026.439956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7027.126623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7032.015510] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7039.277461] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7045.024780] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7051.103444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7051.344629] LustreError: 186277:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7051.353164] Lustre: 186277:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.112@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7051.492553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7058.468692] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7066.198810] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7068.128125] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7068.130086] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7068.133171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7068.171202] Lustre: Skipped 2 previous similar messages [ 7073.256482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7073.266499] Lustre: Skipped 4 previous similar messages [ 7074.400959] Lustre: server umount lustre-MDT0000 complete [ 7078.378572] LustreError: 184215:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7078.423563] LustreError: 184215:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7078.475206] LustreError: 185197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789394985 with bad export cookie 2599396004960730834 [ 7078.478994] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7078.489084] LustreError: 185197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7078.904713] Lustre: server umount lustre-MDT0001 complete [ 7093.304933] Lustre: server umount lustre-OST0000 complete [ 7102.563810] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 7114.482935] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 7151.815993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7152.040973] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7152.059694] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7152.110886] Lustre: lustre-MDT0000: new disk, initializing [ 7152.200464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7152.224485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7156.966560] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7169.221376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7169.372161] Lustre: 189803:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7169.454631] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7169.458339] Lustre: Skipped 1 previous similar message [ 7169.569150] Lustre: lustre-MDT0001: new disk, initializing [ 7169.674775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7169.714942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7169.729178] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7175.164168] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7185.998787] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7192.204840] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7199.397822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7199.634673] Lustre: lustre-OST0000: new disk, initializing [ 7199.638374] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7199.643553] Lustre: 191822:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7199.719615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7205.914524] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7206.539299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7206.559753] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7206.695213] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7217.501918] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7222.922718] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7223.157492] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7228.300991] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7228.507457] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7232.496222] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7232.508738] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7232.523540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7232.529850] Lustre: Skipped 1 previous similar message [ 7236.705492] Lustre: server umount lustre-OST0000 complete [ 7241.698190] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7241.715406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7241.746142] Lustre: Skipped 1 previous similar message [ 7241.761271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7247.477132] Lustre: server umount lustre-MDT0000 complete [ 7247.843704] LustreError: 190916:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7247.876810] LustreError: 190916:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7251.864133] LustreError: 189796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789395158 with bad export cookie 2599396004960731562 [ 7251.869504] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7251.879460] LustreError: 189796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7252.140183] Lustre: server umount lustre-MDT0001 complete [ 7261.342751] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 10:12:46 (1789395166) [ 7282.521193] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 7284.935598] Key type lgssc unregistered [ 7285.191981] LNet: 194912:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7285.196711] LNetError: 194912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7285.205710] LNet: Removed LNI 192.168.201.112@tcp [ 7286.467604] Key type .llcrypt unregistered [ 7286.471521] Key type ._llcrypt unregistered [ 7307.925690] Key type ._llcrypt registered [ 7307.928862] Key type .llcrypt registered [ 7308.155706] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 7309.724285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7309.835930] alg: No test for adler32 (adler32-zlib) [ 7310.879403] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [ 7311.200157] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 7312.967318] Key type lgssc registered [ 7314.070891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7321.945596] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7321.961683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7323.313751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7327.158277] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7330.404435] LustreError: 196598:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 7332.099238] Lustre: New target registration enabled indefinitely [ 7332.838052] LustreError: 196127:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7332.858288] LustreError: 196127:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7337.956335] LustreError: 196128:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7340.291913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7340.588948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7344.849332] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7351.077336] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7356.378231] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7363.762201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7363.909819] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7363.912412] LustreError: 198235:0:(llog.c:1696:llog_backup()) MGC192.168.201.112@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7363.917749] Lustre: 198235:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.112@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7364.010670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7369.980242] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7376.389129] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7378.402049] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7378.416870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7378.427902] Lustre: Skipped 1 previous similar message [ 7378.437116] Lustre: Skipped 1 previous similar message [ 7383.522499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7383.534174] Lustre: Skipped 3 previous similar messages [ 7384.800576] Lustre: server umount lustre-MDT0000 complete [ 7388.590132] LustreError: 196108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789395295 with bad export cookie 13986368035933467072 [ 7388.595367] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7388.599360] LustreError: 196108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7388.642420] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7388.656666] LustreError: 198260:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7388.657138] Lustre: Skipped 1 previous similar message [ 7388.659478] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7388.930562] Lustre: server umount lustre-MDT0001 complete [ 7395.452394] Lustre: server umount lustre-OST0000 complete [ 7404.279781] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 7414.391246] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 7450.323732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7450.598993] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7450.625726] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7450.705609] Lustre: lustre-MDT0000: new disk, initializing [ 7450.789966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7450.805302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7454.972451] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7466.132798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7466.283615] Lustre: 201759:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7466.331250] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7466.333920] Lustre: Skipped 1 previous similar message [ 7466.336857] LustreError: 202589:0:(llog.c:1696:llog_backup()) MGC192.168.201.112@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7466.342486] LustreError: 202589:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7466.346300] Lustre: 202589:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.112@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7466.360777] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7466.364334] Lustre: Skipped 1 previous similar message [ 7466.427779] Lustre: lustre-MDT0001: new disk, initializing [ 7466.567577] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7466.612615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7466.629676] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7471.522686] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7479.721273] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7484.752533] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7490.748530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7490.943578] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7490.945438] Lustre: Skipped 1 previous similar message [ 7490.947256] LustreError: 203751:0:(llog.c:1696:llog_backup()) MGC192.168.201.112@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7490.950447] LustreError: 203751:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7490.952720] Lustre: 203751:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.112@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7490.956475] Lustre: 203751:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7490.971245] Lustre: lustre-OST0000: new disk, initializing [ 7490.980808] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7491.003667] Lustre: 203777:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7491.108462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7497.471935] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7500.874037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7500.890193] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7501.011082] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7506.830417] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7512.556690] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7512.771719] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7518.132444] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7518.377333] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7521.760034] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7521.762551] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7521.768196] LustreError: Skipped 1 previous similar message [ 7521.770058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7521.778586] Lustre: Skipped 1 previous similar message [ 7521.795154] Lustre: Skipped 1 previous similar message [ 7526.576534] Lustre: server umount lustre-OST0000 complete [ 7531.103377] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7531.110957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7531.125681] Lustre: Skipped 1 previous similar message [ 7531.147144] Lustre: Skipped 1 previous similar message [ 7537.630331] Lustre: server umount lustre-MDT0000 complete [ 7540.966537] LustreError: 201752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789395447 with bad export cookie 13986368035933467793 [ 7540.968018] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7540.979050] LustreError: 201752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7541.363058] Lustre: server umount lustre-MDT0001 complete [ 7549.991958] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 10:17:34 (1789395454) [ 7566.914615] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 7577.496721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7577.906754] LustreError: 206904:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7577.945113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7582.426088] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7583.202457] LustreError: 206905:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7591.667554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7596.523893] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7599.146630] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7605.482205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7612.248875] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7620.688684] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7620.821174] Lustre: lustre-OST0001: new disk, initializing [ 7620.825840] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7620.837453] Lustre: 209261:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7620.966269] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7620.970983] Lustre: Skipped 2 previous similar messages [ 7626.980569] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7627.369716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7627.382565] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7627.499054] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7636.804658] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7641.910771] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7643.352925] Lustre: DEBUG MARKER: ADD OST3 [ 7647.654101] loop: module loaded [ 7647.694391] loop0: detected capacity change from 0 to 204800000 [ 7647.936496] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7647.948769] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7647.967309] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7654.012679] loop0: detected capacity change from 0 to 204800000 [ 7655.230124] Lustre: DEBUG MARKER: STOP OST3 [ 7660.709044] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7661.049507] Lustre: lustre-OST0003: new disk, initializing [ 7661.067032] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7661.075533] Lustre: 211248:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7662.476436] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7662.671949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7662.686628] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7662.752522] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7664.480427] LustreError: 211376:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7665.563507] Lustre: Failing over lustre-OST0003 [ 7665.884076] Lustre: server umount lustre-OST0003 complete [ 7667.720379] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7667.729019] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7667.737462] Lustre: Skipped 1 previous similar message [ 7667.738808] LustreError: 208442:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7667.738819] LustreError: 208442:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7669.546550] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7678.325859] Lustre: DEBUG MARKER: START OST3 again [ 7684.010170] LustreError: 209648:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7684.054163] LustreError: 209648:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 7686.909322] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7687.403742] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7687.412703] Lustre: Skipped 1 previous similar message [ 7687.438666] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7688.831920] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7689.023943] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7689.024533] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7693.664145] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7712.921701] Lustre: Failing over lustre-OST0003 [ 7712.997658] Lustre: server umount lustre-OST0003 complete [ 7713.277870] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7718.375053] LustreError: 209648:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7718.391422] LustreError: 209648:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7730.146082] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7730.151264] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7730.161319] Lustre: Skipped 1 previous similar message [ 7730.168425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7730.172367] Lustre: Skipped 2 previous similar messages [ 7736.612479] Lustre: server umount lustre-MDT0000 complete [ 7741.115954] LustreError: 208087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789395647 with bad export cookie 13986368035933468675 [ 7741.127330] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7741.143747] LustreError: 208087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7746.977870] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7746.983648] Lustre: Skipped 7 previous similar messages [ 7747.872679] Lustre: server umount lustre-MDT0001 complete [ 7753.500908] Lustre: server umount lustre-OST0000 complete [ 7757.789675] Lustre: server umount lustre-OST0001 complete [ 7764.114306] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 7773.079827] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 7813.116508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7813.346771] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7813.377385] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7813.454860] Lustre: lustre-MDT0000: new disk, initializing [ 7813.546724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7818.499836] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7830.007912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7830.111747] Lustre: 216719:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7830.159631] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7830.162045] Lustre: Skipped 1 previous similar message [ 7830.214805] Lustre: lustre-MDT0001: new disk, initializing [ 7830.296160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7830.302592] Lustre: Skipped 1 previous similar message [ 7830.348108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7830.363077] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7835.091444] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7844.325249] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7850.300639] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7856.684952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7856.932897] Lustre: 218734:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7858.972393] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7859.128074] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7863.617581] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7872.915696] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7877.468848] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7877.671238] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7881.848127] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7881.978065] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7884.773570] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7884.777482] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7884.785910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7884.797666] Lustre: Skipped 6 previous similar messages [ 7884.815074] Lustre: Skipped 1 previous similar message [ 7889.894260] LustreError: 218878:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7889.920542] LustreError: 218878:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [ 7890.032316] Lustre: server umount lustre-OST0000 complete [ 7908.319219] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7908.480777] Lustre: server umount lustre-MDT0000 complete [ 7912.736775] LustreError: 216712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789395819 with bad export cookie 13986368035933470971 [ 7912.753765] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7913.084984] Lustre: server umount lustre-MDT0001 complete [ 7917.387471] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 10:23:41 (1789395821) [ 7934.715273] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 7943.697713] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 7987.626351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7988.125579] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7988.162721] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7988.168291] Lustre: Skipped 1 previous similar message [ 7988.292368] Lustre: lustre-MDT0000: new disk, initializing [ 7988.295438] Lustre: Skipped 1 previous similar message [ 7988.377786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7988.388918] Lustre: Skipped 1 previous similar message [ 7993.084522] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8007.333439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8007.445301] Lustre: 223347:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8007.786762] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8012.640680] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8023.145652] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8030.426151] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8039.819547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8040.360803] Lustre: 225364:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8041.817118] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8041.986144] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8047.349310] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8059.440286] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8065.820356] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8066.099813] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8071.007630] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8071.183695] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8077.794184] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8077.811621] LustreError: Skipped 1 previous similar message [ 8077.813399] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8077.839448] Lustre: Skipped 2 previous similar messages [ 8077.847724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8077.850519] Lustre: Skipped 6 previous similar messages [ 8079.609883] Lustre: server umount lustre-OST0000 complete [ 8091.091718] Lustre: server umount lustre-MDT0000 complete [ 8093.152227] LustreError: 223360:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8093.183485] LustreError: 223360:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8095.160101] LustreError: 223340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789396001 with bad export cookie 13986368035933471853 [ 8095.165840] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8095.168579] LustreError: 223340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8095.637758] Lustre: server umount lustre-MDT0001 complete [ 8105.541555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8106.111484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8106.117279] Lustre: Skipped 4 previous similar messages [ 8111.128552] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8122.006674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8127.100214] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8135.518910] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8142.782565] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8150.300548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8158.766281] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8168.069810] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8176.901975] LustreError: 230257:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 8182.755357] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8182.775840] LustreError: Skipped 1 previous similar message [ 8183.046692] Lustre: server umount lustre-OST0000 complete [ 8202.207129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8202.518597] Lustre: server umount lustre-MDT0000 complete [ 8206.055396] LustreError: 227564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789396112 with bad export cookie 13986368035933472735 [ 8206.063665] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8206.073567] LustreError: 227564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8206.320704] Lustre: server umount lustre-MDT0001 complete [ 8307.797498] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 8310.938838] Key type lgssc unregistered [ 8311.182931] LNet: 231439:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8311.187274] LNetError: 231439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8311.208333] LNet: Removed LNI 192.168.201.112@tcp [ 8312.016262] Key type .llcrypt unregistered [ 8312.017528] Key type ._llcrypt unregistered [ 8338.143555] Key type ._llcrypt registered [ 8338.146165] Key type .llcrypt registered [ 8338.260711] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 8354.080689] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 8355.322421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8355.402279] alg: No test for adler32 (adler32-zlib) [ 8356.927472] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [ 8357.293341] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 8359.034229] Key type lgssc registered [ 8360.837340] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8403.390816] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8403.441340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8404.616613] LustreError: 234794:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) lustre: New filesystem registration disabled. Use 'lctl set_param allow_register=unlimited' to allow new filesystem registrations: rc = -13 [ 8404.626995] Lustre: First target registration detected: opening a one-time 24-hour registration window. Use 'lctl set_param allow_register=0' to close it early, or 'lctl set_param -P allow_register=0' to lock down permanently. [ 8404.665896] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8404.695174] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8404.747744] Lustre: lustre-MDT0000: new disk, initializing [ 8404.813190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8404.828218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8408.908768] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8414.897538] LustreError: 235328:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 8416.767319] Lustre: New target registration enabled indefinitely [ 8427.064477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8427.273454] Lustre: 234793:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8427.348297] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8427.351635] Lustre: Skipped 1 previous similar message [ 8427.448958] Lustre: lustre-MDT0001: new disk, initializing [ 8427.610035] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8427.683808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8427.695406] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8432.198920] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8442.608740] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8448.720687] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8456.068424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8456.355775] Lustre: lustre-OST0000: new disk, initializing [ 8456.359149] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8456.364278] Lustre: 237008:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8456.446621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8463.461868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8463.476306] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8463.669789] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8463.834727] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8473.820918] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8478.313561] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8478.559240] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8482.906656] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8483.157856] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8489.443967] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8489.447189] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8489.458304] LustreError: Skipped 1 previous similar message [ 8489.460529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8489.471442] Lustre: Skipped 1 previous similar message [ 8489.489196] Lustre: Skipped 1 previous similar message [ 8491.491544] Lustre: server umount lustre-OST0000 complete [ 8499.169980] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8499.196420] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8499.215214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8501.912198] Lustre: server umount lustre-MDT0000 complete [ 8506.878306] LustreError: 234784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789396413 with bad export cookie 13823594968635157928 [ 8506.885083] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8506.899543] LustreError: 234784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8507.270986] Lustre: server umount lustre-MDT0001 complete [ 8530.154424] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 8542.931483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8543.428470] LustreError: 239943:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8543.447894] LustreError: 239943:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8543.569098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8548.832603] LustreError: 239944:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8550.407357] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8553.957077] LustreError: 239943:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8559.089157] LustreError: 239944:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8561.807052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8562.342301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8567.408903] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8570.336876] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8578.134045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8578.535639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8586.776413] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8595.869125] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8595.997507] Lustre: lustre-OST0001: new disk, initializing [ 8596.000294] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8596.004255] Lustre: 242300:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8596.097156] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8602.378352] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8605.280771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8605.295391] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8605.396179] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8613.409306] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8620.223544] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8631.946934] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 10:35:36 (1789396536) [ 8636.386949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8636.393099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8636.402895] Lustre: Skipped 2 previous similar messages [ 8636.425464] Lustre: Skipped 4 previous similar messages [ 8640.443249] Lustre: server umount lustre-MDT0000 complete [ 8641.504872] LustreError: 239943:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8641.529721] LustreError: 239943:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8644.239974] LustreError: 239925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789396550 with bad export cookie 13823594968635158810 [ 8644.259921] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8644.261065] LustreError: 239925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8644.508720] Lustre: server umount lustre-MDT0001 complete [ 8658.527824] Lustre: server umount lustre-OST0000 complete [ 8673.287758] Lustre: server umount lustre-OST0001 complete [ 8681.436269] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 8692.406323] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 8731.894713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8732.164968] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8732.207379] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8732.279570] Lustre: lustre-MDT0000: new disk, initializing [ 8732.368481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8732.385545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8736.191652] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8747.559080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8747.679408] Lustre: 246249:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8747.737992] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8747.741478] Lustre: Skipped 1 previous similar message [ 8747.818056] Lustre: lustre-MDT0001: new disk, initializing [ 8747.967739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8747.978718] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8753.256481] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8763.266555] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8769.619487] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8776.253363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8776.607491] Lustre: lustre-OST0000: new disk, initializing [ 8776.611518] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8776.622051] Lustre: 248264:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8778.442875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8778.459370] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8778.636866] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8783.568161] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8793.485827] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8798.450892] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8798.668867] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8803.279387] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8803.474565] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8809.457563] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8809.459772] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8809.475882] Lustre: Skipped 3 previous similar messages [ 8809.479865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8809.487167] LustreError: Skipped 1 previous similar message [ 8809.507147] Lustre: Skipped 1 previous similar message [ 8814.570274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8814.576681] Lustre: Skipped 1 previous similar message [ 8820.198190] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8820.518410] Lustre: server umount lustre-OST0000 complete [ 8824.805329] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8824.807802] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8824.824777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8824.837162] Lustre: Skipped 3 previous similar messages [ 8831.000371] Lustre: server umount lustre-MDT0000 complete [ 8835.041937] LustreError: 246261:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8835.069156] LustreError: 246261:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8835.500736] LustreError: 246242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789396742 with bad export cookie 13823594968635160945 [ 8835.513869] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8835.514729] LustreError: 246242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8835.999742] Lustre: server umount lustre-MDT0001 complete [ 8845.501423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8845.972041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8845.983296] Lustre: Skipped 2 previous similar messages [ 8850.913914] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8851.428133] LustreError: 250483:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8851.451964] LustreError: 250483:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8860.100748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8865.643940] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8873.524522] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8879.131194] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8886.143623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8893.290407] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8900.435645] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9045.586109] LustreError: 253099:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 9049.569313] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9049.582633] Lustre: Skipped 1 previous similar message [ 9049.589058] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9049.593798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9049.610589] Lustre: Skipped 3 previous similar messages [ 9055.687987] Lustre: server umount lustre-OST0000 complete [ 9065.271628] Lustre: server umount lustre-MDT0000 complete [ 9069.054888] LustreError: 250464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789396975 with bad export cookie 13823594968635161820 [ 9069.057462] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9069.066841] LustreError: 250464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9069.492690] Lustre: server umount lustre-MDT0001 complete [ 9171.029171] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 9173.944605] Key type lgssc unregistered [ 9174.197512] LNet: 254283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9174.203731] LNetError: 254283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9174.224751] LNet: Removed LNI 192.168.201.112@tcp [ 9175.111258] Key type .llcrypt unregistered [ 9175.116692] Key type ._llcrypt unregistered [ 9198.502450] Key type ._llcrypt registered [ 9198.504485] Key type .llcrypt registered [ 9198.642294] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 9214.432236] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 9215.866608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9215.966517] alg: No test for adler32 (adler32-zlib) [ 9217.040359] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [ 9217.359630] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 9219.071232] Key type lgssc registered [ 9220.171703] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9264.568234] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 9274.752435] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9274.768600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9276.064260] LustreError: 258351:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) lustre: New filesystem registration disabled. Use 'lctl set_param allow_register=unlimited' to allow new filesystem registrations: rc = -13 [ 9276.087736] Lustre: First target registration detected: opening a one-time 24-hour registration window. Use 'lctl set_param allow_register=0' to close it early, or 'lctl set_param -P allow_register=0' to lock down permanently. [ 9276.124899] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9276.184898] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9276.292872] Lustre: lustre-MDT0000: new disk, initializing [ 9276.376452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9276.414695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9281.779795] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9287.397740] LustreError: 258882:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 9289.351161] Lustre: New target registration enabled indefinitely [ 9298.550883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9298.655282] Lustre: 258351:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9298.697620] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9298.700978] Lustre: Skipped 1 previous similar message [ 9298.762568] Lustre: lustre-MDT0001: new disk, initializing [ 9298.849386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9298.892511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9298.906086] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9304.373818] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9309.122427] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9315.463577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9315.697214] Lustre: lustre-OST0000: new disk, initializing [ 9315.701867] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9315.716180] Lustre: 260222:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9315.835789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9322.561834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9322.591540] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9322.785598] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9323.036212] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9333.875046] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9334.087051] Lustre: lustre-OST0001: new disk, initializing [ 9334.091234] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9334.097106] Lustre: 261096:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9334.199578] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9341.011430] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9342.089544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9342.110820] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9342.239683] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9351.261945] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9360.067387] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9373.162280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9373.178659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9373.181132] Lustre: Skipped 3 previous similar messages [ 9373.204646] Lustre: Skipped 3 previous similar messages [ 9377.724779] Lustre: server umount lustre-MDT0000 complete [ 9381.172397] LustreError: 258340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789397287 with bad export cookie 2836519277756469881 [ 9381.175164] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9381.181465] LustreError: 258340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9381.461870] Lustre: server umount lustre-MDT0001 complete [ 9394.629523] Lustre: server umount lustre-OST0000 complete [ 9409.035857] Lustre: server umount lustre-OST0001 complete [ 9424.297451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9434.516774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9443.490928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9452.144150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9472.743955] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 9484.663175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9484.690103] Lustre: lustre-MDT0000: reset Object Index mappings [ 9485.169803] LustreError: 266108:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9485.191467] LustreError: 266108:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 9485.279403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9490.410192] LustreError: 266109:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9490.591449] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9494.566550] LustreError: 266108:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9499.620058] LustreError: 266109:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9500.711108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9500.730538] Lustre: lustre-MDT0001: reset Object Index mappings [ 9501.122228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9506.004589] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9509.631650] Lustre: 267300:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9517.009860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9517.400988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9523.585788] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9526.650274] LustreError: 267654:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9529.726388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9532.768681] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9538.040831] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9539.066636] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9548.078635] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9551.691966] Lustre: 269169:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9557.472746] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9557.482670] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9557.501604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9558.499592] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9558.503967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9558.517970] Lustre: Skipped 1 previous similar message [ 9558.537693] Lustre: Skipped 3 previous similar messages [ 9562.120316] Lustre: server umount lustre-MDT0000 complete [ 9563.623737] LustreError: 266109:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9563.647342] LustreError: 266109:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9565.861656] LustreError: 266091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789397472 with bad export cookie 2836519277756476671 [ 9565.864059] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9565.873908] LustreError: 266091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9566.177713] Lustre: server umount lustre-MDT0001 complete [ 9579.941285] Lustre: server umount lustre-OST0000 complete [ 9592.907700] Lustre: server umount lustre-OST0001 complete [ 9611.316236] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 10:51:56 (1789397516) [ 9628.916181] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 9640.541987] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 9678.521767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9678.841495] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9678.890099] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9678.956654] Lustre: lustre-MDT0000: new disk, initializing [ 9679.060990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9679.067132] Lustre: Skipped 1 previous similar message [ 9679.089609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9684.134817] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9698.147320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9698.291295] Lustre: 273425:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9698.353882] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9698.364170] Lustre: Skipped 1 previous similar message [ 9698.469985] Lustre: lustre-MDT0001: new disk, initializing [ 9698.610187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9698.621308] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9703.801855] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9712.713450] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9718.041987] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9724.380255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9724.661205] Lustre: lustre-OST0000: new disk, initializing [ 9724.664600] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9724.670519] Lustre: 275443:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9726.750189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9726.762278] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9726.865529] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9730.221190] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9738.091519] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9742.566535] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9742.724846] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9747.066586] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9747.260492] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9752.544725] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9752.551543] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9752.563973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9752.570727] Lustre: Skipped 2 previous similar messages [ 9757.669642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9757.680103] Lustre: Skipped 2 previous similar messages [ 9763.295163] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9763.589418] Lustre: server umount lustre-OST0000 complete [ 9767.908473] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9767.914889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9767.939523] Lustre: Skipped 1 previous similar message [ 9767.956084] Lustre: Skipped 3 previous similar messages [ 9773.558331] Lustre: server umount lustre-MDT0000 complete [ 9778.068505] LustreError: 275442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789397684 with bad export cookie 2836519277756478771 [ 9778.088850] LustreError: 275442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9778.095716] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9778.596402] Lustre: server umount lustre-MDT0001 complete [ 9799.211590] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 9809.457313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9809.942537] LustreError: 278378:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9810.003049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9810.013751] Lustre: Skipped 2 previous similar messages [ 9814.814477] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9825.900281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9831.035779] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9834.191737] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9840.996330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9847.485941] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9856.564856] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9856.898960] Lustre: lustre-OST0001: new disk, initializing [ 9856.906914] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9856.917344] Lustre: 280733:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9858.704885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9858.719834] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9858.873101] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9864.604777] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9875.396801] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9879.404413] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9886.054428] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9886.346709] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9897.956667] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9897.963180] LustreError: Skipped 1 previous similar message [ 9897.969132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9897.992663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9897.995151] Lustre: Skipped 2 previous similar messages [ 9901.917691] Lustre: server umount lustre-MDT0000 complete [ 9903.010303] LustreError: 278378:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9903.055063] LustreError: 278378:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 9906.587538] LustreError: 278360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789397813 with bad export cookie 2836519277756479653 [ 9906.596058] LustreError: 278360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9906.596320] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9907.192935] Lustre: server umount lustre-MDT0001 complete [ 9922.038637] Lustre: server umount lustre-OST0000 complete [ 9923.936541] Lustre: 255855:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789397814/real 1789397814] req@ffff8c6d4cce1180 x1876318877883904/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789397830 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9923.968842] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9923.986329] Lustre: Skipped 3 previous similar messages [ 9926.498472] Lustre: server umount lustre-OST0001 complete [ 9945.401129] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 9959.886387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9960.315476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9960.326693] Lustre: Skipped 3 previous similar messages [ 9966.156600] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9969.644568] LustreError: 284075:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9969.695573] LustreError: 284075:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9976.689678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9981.841371] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9985.079874] Lustre: 285263:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9991.990914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9998.263631] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10004.805722] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10008.029898] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10013.715966] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10014.671812] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10022.344729] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10025.762722] Lustre: 287141:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10030.077508] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10030.083050] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10041.795534] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 10:59:06 (1789397946) [10058.731241] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10058.748042] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10058.761746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10058.772361] Lustre: Skipped 4 previous similar messages [10063.780845] Lustre: server umount lustre-MDT0000 complete [10068.530877] LustreError: 287144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789397975 with bad export cookie 2836519277756486058 [10068.536669] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10068.540967] LustreError: 287144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10075.010770] Lustre: server umount lustre-MDT0001 complete [10085.509153] Lustre: server umount lustre-OST0000 complete [10089.579484] Lustre: server umount lustre-OST0001 complete [10097.554875] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [10107.810665] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [10156.375603] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [10166.246602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10166.497778] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10166.516067] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10166.580514] Lustre: lustre-MDT0000: new disk, initializing [10166.667108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10172.270768] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10186.488506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10186.730927] Lustre: 291685:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10187.116879] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10192.624718] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10198.770996] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10204.617547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10204.888120] Lustre: 293367:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10206.513991] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10211.560713] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10222.732039] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10223.085599] Lustre: 294239:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10223.222357] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10223.233880] Lustre: Skipped 6 previous similar messages [10224.577062] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10224.580342] Lustre: Skipped 1 previous similar message [10224.681258] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10229.617503] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10240.414634] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10260.460223] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 11:02:45 (1789398165) [10264.037755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10264.041458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10264.057259] Lustre: Skipped 5 previous similar messages [10264.061864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10264.065380] Lustre: Skipped 8 previous similar messages [10265.507888] Lustre: server umount lustre-MDT0000 complete [10265.569557] LustreError: 291697:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10265.591714] LustreError: 291697:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [10270.133496] LustreError: 291677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789398176 with bad export cookie 2836519277756493982 [10270.140996] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10270.146570] LustreError: 291677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10270.729499] Lustre: server umount lustre-MDT0001 complete [10284.323443] Lustre: server umount lustre-OST0000 complete [10286.563710] Lustre: 255855:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789398177/real 1789398177] req@ffff8c6e5018ed80 x1876318878036096/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789398193 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10287.650355] Lustre: server umount lustre-OST0001 complete [10295.202634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10299.917404] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10308.802709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10314.131300] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10321.810616] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10328.429533] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10337.316314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10345.551282] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10353.066819] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10361.517805] Lustre: New target registration disabled [10372.858598] loop1: detected capacity change from 0 to 204800000 [10373.147278] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10373.164694] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10373.192185] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10377.776815] loop1: detected capacity change from 0 to 204800000 [10383.267539] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10383.343317] LustreError: 296473:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs20363: New filesystem registration disabled. Use 'lctl set_param allow_register=unlimited' to allow new filesystem registrations: rc = -13 [10383.355293] LustreError: 296473:0:(mgs_llog.c:736:mgs_check_index()) Can't get db for fs20363 [10383.361526] LustreError: 296473:0:(mgs_handler.c:525:mgs_target_reg()) fs20363-MDT0001: Error checking target index: rc = -13 [10383.374392] LustreError: fs20363-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10383.382619] LustreError: 300253:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [10383.394155] LustreError: 300253:0:(tgt_mount.c:2128:server_put_super()) no obd fs20363-MDT0001 [10383.403454] LustreError: 300253:0:(tgt_mount.c:116:server_deregister_mount()) fs20363-MDT0001 not registered [10383.440610] Lustre: server umount fs20363-MDT0001 complete [10383.443503] LustreError: 300253:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10385.304552] Lustre: New target registration enabled indefinitely [10391.255434] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10391.397748] Lustre: Setting parameter fs20363-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs20363-MDT0001 [10391.406843] Lustre: Skipped 1 previous similar message [10391.443244] Lustre: srv-fs20363-MDT0001: No data found on store. Initialize space. [10391.446740] Lustre: Skipped 4 previous similar messages [10391.515860] Lustre: fs20363-MDT0001: new disk, initializing [10391.518827] Lustre: Skipped 3 previous similar messages [10391.650087] Lustre: srv-fs20363-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10391.665175] Lustre: 300720:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs20363-MDT0001-mdtlov: can't create new llog: rc = -115 [10391.678197] LustreError: 300720:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs20363-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10396.037058] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10404.027374] Lustre: server umount fs20363-MDT0001 complete [10413.973657] Lustre: New target registration disabled [10418.682348] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10419.990401] loop1: detected capacity change from 0 to 204800000 [10426.772804] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10427.066933] Lustre: 302289:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs20363-MDT0001-mdtlov: can't create new llog: rc = -115 [10427.073593] LustreError: 302289:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs20363-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10431.387268] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10437.732976] Lustre: server umount fs20363-MDT0001 complete [10450.910108] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.12@tcp (stopping) [10452.449268] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10452.464117] LustreError: Skipped 1 previous similar message [10452.471283] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10452.500615] Lustre: Skipped 4 previous similar messages [10464.226554] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10464.435964] Lustre: server umount lustre-MDT0000 complete [10473.167727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10473.330967] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10479.095348] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10479.231058] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10488.266243] loop1: detected capacity change from 0 to 204800000 [10493.485031] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10493.804356] Lustre: 304785:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs20363-MDT0001-mdtlov: can't create new llog: rc = -115 [10493.809231] LustreError: 304785:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs20363-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10498.556544] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10503.786798] Lustre: server umount fs20363-MDT0001 complete [10524.499531] Lustre: New target registration enabled indefinitely [10525.168487] LustreError: 298360:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10525.202981] LustreError: 298360:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 112 previous similar messages [10528.225445] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10532.457515] Lustre: server umount lustre-OST0000 complete [10542.699928] Lustre: server umount lustre-MDT0000 complete [10546.221782] LustreError: 298386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789398452 with bad export cookie 2836519277756496663 [10546.226275] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10546.229172] LustreError: 298386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10546.550671] Lustre: server umount lustre-MDT0001 complete [10558.892836] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [10561.842540] Key type lgssc unregistered [10562.387342] LNet: 307227:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10562.395272] LNetError: 307227:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10562.404665] LNet: Removed LNI 192.168.201.112@tcp [10563.200212] Key type .llcrypt unregistered [10563.206823] Key type ._llcrypt unregistered [10585.770294] Key type ._llcrypt registered [10585.771617] Key type .llcrypt registered [10585.855568] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [10588.138056] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10588.162230] alg: No test for adler32 (adler32-zlib) [10589.383534] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [10589.710558] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [10591.503437] Key type lgssc registered [10592.422943] Lustre: Echo OBD driver; http://www.lustre.org/ [10600.246921] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10600.263793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10601.706100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10606.189075] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10611.992640] Lustre: New target registration enabled indefinitely [10612.197445] LustreError: 308497:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10612.235094] LustreError: 308497:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10617.319680] LustreError: 308498:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10620.123185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10620.621655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10626.305901] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10635.423996] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10643.510912] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10650.661935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10651.094607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10659.915209] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10665.451051] LustreError: 310572:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10667.372617] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10670.568022] LustreError: 310573:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10674.680741] LustreError: 310570:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10674.696264] LustreError: 310570:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10684.846689] LustreError: 311195:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10684.877294] LustreError: 311195:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [10693.570858] loop1: detected capacity change from 0 to 204800000 [10693.818421] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10693.827838] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10693.845081] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10699.116540] loop1: detected capacity change from 0 to 204800000 [10702.314197] LustreError: 310570:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10702.332652] LustreError: 310570:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [10704.789463] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10704.948161] Lustre: fs5121-OST0000: new disk, initializing [10704.952535] Lustre: srv-fs5121-OST0000: No data found on store. Initialize space. [10704.961345] Lustre: 312494:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10705.049333] Lustre: fs5121-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10711.766547] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10719.292795] Lustre: New target registration disabled [10721.344638] Lustre: server umount fs5121-OST0000 complete [10735.564575] loop1: detected capacity change from 0 to 204800000 [10736.029060] LustreError: 312355:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10736.063890] LustreError: 312355:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 18 previous similar messages [10741.658761] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10741.817175] Lustre: fs5121-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10743.280193] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10748.396828] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10748.428313] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10757.180190] loop2: detected capacity change from 0 to 204800000 [10757.482202] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10757.494583] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10757.507707] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10764.120107] loop2: detected capacity change from 0 to 204800000 [10770.140185] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10770.283097] LustreError: fs5121-OST0001: New target registration disabled. Use 'lctl set_param allow_register=unlimited' to allow new target registrations. [10770.295439] LustreError: fs5121-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10770.302982] LustreError: 315534:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [10770.307972] LustreError: 315534:0:(tgt_mount.c:2128:server_put_super()) no obd fs5121-OST0001 [10770.317267] LustreError: 315534:0:(tgt_mount.c:116:server_deregister_mount()) fs5121-OST0001 not registered [10770.380719] Lustre: server umount fs5121-OST0001 complete [10770.384230] LustreError: 315534:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10775.184269] Lustre: New target registration enabled indefinitely [10781.699884] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10781.810432] Lustre: fs5121-OST0001: new disk, initializing [10781.815096] Lustre: srv-fs5121-OST0001: No data found on store. Initialize space. [10781.823668] Lustre: 316087:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10781.918726] Lustre: fs5121-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10788.439889] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10795.387667] Lustre: New target registration disabled [10797.209256] Lustre: server umount fs5121-OST0000 complete [10802.602635] LustreError: 310570:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10802.638123] LustreError: 310570:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 38 previous similar messages [10806.212549] Lustre: server umount fs5121-OST0001 complete [10817.412547] loop1: detected capacity change from 0 to 204800000 [10823.351227] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10823.583336] Lustre: fs5121-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10830.297793] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10836.911526] loop2: detected capacity change from 0 to 204800000 [10842.730414] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10850.443032] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10861.038499] Lustre: server umount fs5121-OST0000 complete [10870.802700] Lustre: server umount fs5121-OST0001 complete [10883.717233] Lustre: New target registration enabled indefinitely [10886.626032] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10886.637752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10886.645325] Lustre: Skipped 1 previous similar message [10886.661066] Lustre: Skipped 1 previous similar message [10891.749715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10891.764690] Lustre: Skipped 1 previous similar message [10891.961595] Lustre: server umount lustre-OST0000 complete [10896.867710] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10896.868446] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10896.875406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10896.891294] Lustre: Skipped 1 previous similar message [10896.900060] Lustre: Skipped 1 previous similar message [10901.993982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10902.010239] Lustre: Skipped 1 previous similar message [10902.369799] Lustre: server umount lustre-MDT0000 complete [10905.958831] LustreError: 308478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789398812 with bad export cookie 2311895682713419369 [10905.967048] LustreError: 308478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10905.967983] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10906.492637] Lustre: server umount lustre-MDT0001 complete [10916.928144] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [10919.347722] Key type lgssc unregistered [10919.593930] LNet: 322109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10919.600385] LNetError: 322109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10919.612785] LNet: Removed LNI 192.168.201.112@tcp [10920.676688] Key type .llcrypt unregistered [10920.681892] Key type ._llcrypt unregistered [10941.811663] Key type ._llcrypt registered [10941.814394] Key type .llcrypt registered [10941.897142] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [10943.362380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10943.383224] alg: No test for adler32 (adler32-zlib) [10944.764868] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [10945.074280] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [10946.759219] Key type lgssc registered [10947.981436] Lustre: Echo OBD driver; http://www.lustre.org/ [10954.917824] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10954.950381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10956.342665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10960.567507] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10965.476395] LustreError: 323380:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10965.499367] LustreError: 323380:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10965.839757] Lustre: New target registration enabled indefinitely [10970.606236] LustreError: 323381:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10973.224241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10973.811472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10977.774929] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10985.654205] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10993.809024] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11002.381195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11002.733266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11008.909168] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11016.828271] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11019.108143] LustreError: 325457:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11024.356942] LustreError: 325458:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11031.536227] LustreError: 325457:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11031.555942] LustreError: 325457:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [11041.768356] LustreError: 326126:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11041.776819] LustreError: 326126:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [11047.406718] loop1: detected capacity change from 0 to 204800000 [11047.574945] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11047.585489] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11047.598514] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11054.581502] loop1: detected capacity change from 0 to 204800000 [11058.586569] LustreError: 326126:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11058.608483] LustreError: 326126:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [11061.501427] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11061.840477] Lustre: fs7504-OST0000: new disk, initializing [11061.850439] Lustre: srv-fs7504-OST0000: No data found on store. Initialize space. [11061.866502] Lustre: 327479:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11061.960259] Lustre: fs7504-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11068.019022] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11073.878201] Lustre: Registration timeout window started: 60 seconds. New targets can be registered during this window. [11076.689606] Lustre: DEBUG MARKER: test157c-1789398936-8304 [11092.968900] LustreError: 325457:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11092.984131] LustreError: 325457:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 18 previous similar messages [11098.093104] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11103.210794] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11133.921859] Lustre: New MDT/OST registration window disabled. Use 'lctl set_param allow_register=' to re-enable (default unit: hours). [11141.457875] loop2: detected capacity change from 0 to 204800000 [11141.672112] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11141.683383] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11141.708338] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11146.597303] loop2: detected capacity change from 0 to 204800000 [11151.565302] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11151.686800] LustreError: fs7504-OST0001: New target registration disabled. Use 'lctl set_param allow_register=unlimited' to allow new target registrations. [11151.693690] LustreError: fs7504-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11151.699572] LustreError: 330642:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [11151.704504] LustreError: 330642:0:(tgt_mount.c:2128:server_put_super()) no obd fs7504-OST0001 [11151.709627] LustreError: 330642:0:(tgt_mount.c:116:server_deregister_mount()) fs7504-OST0001 not registered [11151.734710] Lustre: server umount fs7504-OST0001 complete [11151.737509] LustreError: 330642:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11154.828617] Lustre: New target registration enabled indefinitely [11159.539404] LustreError: 325457:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11159.561859] LustreError: 325457:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 39 previous similar messages [11161.478667] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11161.692570] Lustre: fs7504-OST0001: new disk, initializing [11161.695463] Lustre: srv-fs7504-OST0001: No data found on store. Initialize space. [11161.700034] Lustre: 331197:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11161.764358] Lustre: fs7504-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11170.084496] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11183.105299] Lustre: server umount fs7504-OST0000 complete [11193.437987] Lustre: server umount fs7504-OST0001 complete [11208.672849] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11208.676732] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11208.687058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11210.725788] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11210.726753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11210.740709] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11210.764468] Lustre: Skipped 1 previous similar message [11214.596026] Lustre: server umount lustre-OST0000 complete [11219.439551] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11219.452692] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11219.465317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11224.605229] Lustre: server umount lustre-MDT0000 complete [11228.404821] LustreError: 323362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789399135 with bad export cookie 14607131385492744466 [11228.424513] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11228.770784] Lustre: server umount lustre-MDT0001 complete [11239.497108] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [11242.248699] Key type lgssc unregistered [11242.515389] LNet: 334148:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11242.528327] LNetError: 334148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11242.558481] LNet: Removed LNI 192.168.201.112@tcp [11243.355237] Key type .llcrypt unregistered [11243.359607] Key type ._llcrypt unregistered [11264.081374] Key type ._llcrypt registered [11264.085570] Key type .llcrypt registered [11264.175214] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [11265.324119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11265.373391] alg: No test for adler32 (adler32-zlib) [11266.714809] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [11266.943041] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [11268.615315] Key type lgssc registered [11269.874375] Lustre: Echo OBD driver; http://www.lustre.org/ [11278.440666] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11278.472146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11280.110431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11284.603308] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11289.574712] LustreError: 335467:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11289.589239] LustreError: 335467:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11290.949922] Lustre: New target registration enabled indefinitely [11294.696911] LustreError: 335468:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11299.818097] LustreError: 335468:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11301.376850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11301.933246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11309.390887] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11317.141645] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11322.720962] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11328.827965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11329.254290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11334.642544] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11341.490127] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11346.677048] LustreError: 337544:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11353.566637] LustreError: 337545:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11353.604568] LustreError: 337545:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11356.987783] Lustre: Setting parameter general.allow_register=0 in log params [11363.739464] LustreError: 337544:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11363.760979] LustreError: 337544:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [11364.342188] Lustre: New target registration disabled [11367.862915] LustreError: 338619:0:(mgs_llog.c:6149:mgs_set_param2()) MGS: persistent allow_register only allowed for 0 or unlimited: rc = -22 [11367.881124] LustreError: 338619:0:(mgs_handler.c:1214:mgs_iocontrol()) MGS: setparam err: rc = -22 [11368.812680] LustreError: 338668:0:(mgs_llog.c:6149:mgs_set_param2()) MGS: persistent allow_register only allowed for 0 or unlimited: rc = -22 [11368.817926] LustreError: 338668:0:(mgs_handler.c:1214:mgs_iocontrol()) MGS: setparam err: rc = -22 [11373.536496] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11373.540731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11373.560546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11375.586803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11375.600462] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11375.616466] Lustre: Skipped 1 previous similar message [11378.684349] Lustre: server umount lustre-MDT0000 complete [11380.711659] LustreError: 337545:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11380.730480] LustreError: 337545:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [11381.944136] LustreError: 335448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789399288 with bad export cookie 4292582916690477345 [11381.948714] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11381.959107] LustreError: 335448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11382.199624] Lustre: server umount lustre-MDT0001 complete [11396.124280] Lustre: server umount lustre-OST0000 complete [11409.521394] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [11412.011611] Key type lgssc unregistered [11412.321228] LNet: 339873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11412.329725] LNetError: 339873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11412.359128] LNet: Removed LNI 192.168.201.112@tcp [11413.232187] Key type .llcrypt unregistered [11413.234151] Key type ._llcrypt unregistered [11427.833382] Key type ._llcrypt registered [11427.834532] Key type .llcrypt registered [11427.949837] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [11428.926174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11429.075925] alg: No test for adler32 (adler32-zlib) [11430.150082] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [11430.327411] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [11431.959256] Key type lgssc registered [11432.849890] Lustre: Echo OBD driver; http://www.lustre.org/ [11441.911586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11441.934683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11443.359856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11448.555851] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11452.901326] LustreError: 341110:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11452.932919] LustreError: 341110:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11458.034669] LustreError: 341110:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11458.841268] LustreError: 341142:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [11458.877478] LustreError: 341142:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [11458.906750] LustreError: 341142:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 0, failed: rc = -5 [11459.193273] Lustre: server umount lustre-MDT0000 complete [11475.404805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11498.174746] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [11500.611431] Key type lgssc unregistered [11500.854724] LNet: 343793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11500.858710] LNetError: 343793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11501.926611] LNet: Removed LNI 192.168.201.112@tcp [11503.040521] Key type .llcrypt unregistered [11503.047325] Key type ._llcrypt unregistered [11524.540961] Key type ._llcrypt registered [11524.543969] Key type .llcrypt registered [11524.690755] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [11526.323602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11526.366839] alg: No test for adler32 (adler32-zlib) [11527.675455] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [11528.133831] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [11529.863293] Key type lgssc registered [11531.977859] Lustre: Echo OBD driver; http://www.lustre.org/ [11544.286732] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11544.338391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11545.922744] Lustre: New target registration enabled indefinitely [11545.957853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11551.456379] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11556.336979] LustreError: 345015:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11556.363365] LustreError: 345015:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11561.443247] LustreError: 345016:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11566.572235] LustreError: 345015:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11567.598492] Lustre: Disabling parameter general.allow_register= in log params [11570.658021] LustreError: 345016:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11575.785986] LustreError: 345015:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11576.879991] LustreError: 345048:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [11576.892308] LustreError: 345048:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [11576.899805] LustreError: 345048:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 31, retries 0, failed: rc = -5 [11577.260156] Lustre: server umount lustre-MDT0000 complete [11591.168689] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [11593.806643] Key type lgssc unregistered [11594.095401] LNet: 347032:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11594.103600] LNetError: 347032:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11594.141380] LNet: Removed LNI 192.168.201.112@tcp [11595.243739] Key type .llcrypt unregistered [11595.245077] Key type ._llcrypt unregistered [11616.210882] Key type ._llcrypt registered [11616.213370] Key type .llcrypt registered [11616.356865] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [11617.697939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11617.746559] alg: No test for adler32 (adler32-zlib) [11618.736512] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [11618.915490] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [11620.559325] Key type lgssc registered [11621.875810] Lustre: Echo OBD driver; http://www.lustre.org/ [11630.245285] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11630.266833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11631.739173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11636.968113] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11642.328980] Lustre: New target registration enabled indefinitely [11642.344144] LustreError: 348305:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11642.362902] LustreError: 348305:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11647.463478] LustreError: 348304:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11648.799782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11649.126130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11653.093909] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11660.489737] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11666.007781] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11673.928924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11674.337540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11676.411503] LustreError: 350377:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11681.412282] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11681.787925] LustreError: 350380:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11686.886559] LustreError: 350379:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11689.466992] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11696.541972] LustreError: 350507:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11696.567362] LustreError: 350507:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [11704.303313] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11704.315177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11706.754672] Lustre: server umount lustre-MDT0000 complete [11709.993338] LustreError: 348284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789399616 with bad export cookie 5678564353421008699 [11709.999706] LustreError: 348284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11709.999843] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11710.294536] Lustre: server umount lustre-MDT0001 complete [11723.977814] Lustre: server umount lustre-OST0000 complete [11738.981969] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [11741.484805] Key type lgssc unregistered [11741.742640] LNet: 352294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11741.748262] LNetError: 352294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11741.761507] LNet: Removed LNI 192.168.201.112@tcp [11742.651166] Key type .llcrypt unregistered [11742.653673] Key type ._llcrypt unregistered [11760.250592] Key type ._llcrypt registered [11760.252864] Key type .llcrypt registered [11760.353545] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [11760.989334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11761.051861] alg: No test for adler32 (adler32-zlib) [11762.354557] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [11762.595221] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [11764.319308] Key type lgssc registered [11765.597182] Lustre: Echo OBD driver; http://www.lustre.org/ [11773.567983] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11773.583313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11775.043515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11778.883527] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11785.185399] LustreError: 353513:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11785.201551] LustreError: 353513:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11785.440833] Lustre: Registration timeout window started: 60 seconds. New targets can be registered during this window. [11787.964387] LustreError: 353545:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [11787.972868] LustreError: 353545:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [11787.978082] LustreError: 353545:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 13, retries 0, failed: rc = -5 [11788.354262] Lustre: server umount lustre-MDT0000 complete [11808.917389] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [11812.404195] Key type lgssc unregistered [11812.829709] LNet: 355337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11812.841295] LNetError: 355337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11812.868857] LNet: Removed LNI 192.168.201.112@tcp [11813.951332] Key type .llcrypt unregistered [11813.972507] Key type ._llcrypt unregistered [11835.417336] Key type ._llcrypt registered [11835.419333] Key type .llcrypt registered [11835.561743] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [11837.500210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11837.529381] alg: No test for adler32 (adler32-zlib) [11838.611097] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [11838.777467] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [11840.463172] Key type lgssc registered [11842.084784] Lustre: Echo OBD driver; http://www.lustre.org/ [11853.397264] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11853.433802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11854.963597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11859.365101] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11864.551438] LustreError: 356563:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11864.569884] LustreError: 356563:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11867.522449] Lustre: Setting parameter general.allow_register=unlimited in log params [11869.408749] LustreError: 356595:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [11869.431212] LustreError: 356595:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [11869.437349] LustreError: 356595:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 0, failed: rc = -5 [11869.839837] Lustre: server umount lustre-MDT0000 complete [11890.367556] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [11893.121580] Key type lgssc unregistered [11893.401072] LNet: 358390:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11893.407813] LNetError: 358390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11893.441185] LNet: Removed LNI 192.168.201.112@tcp [11894.256293] Key type .llcrypt unregistered [11894.258393] Key type ._llcrypt unregistered [11910.746172] Key type ._llcrypt registered [11910.748309] Key type .llcrypt registered [11910.837168] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [11911.953487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11911.976793] alg: No test for adler32 (adler32-zlib) [11913.104686] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [11913.523765] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [11915.191316] Key type lgssc registered [11916.297814] Lustre: Echo OBD driver; http://www.lustre.org/ [11925.632965] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11925.659419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11927.056734] Lustre: New target registration enabled indefinitely [11927.093807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11930.971780] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11937.250709] LustreError: 359613:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11937.265518] LustreError: 359613:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11937.446723] Lustre: Disabling parameter general.allow_register= in log params [11942.370980] LustreError: 359612:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11943.386762] Lustre: 360579:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify general/allow_register= (mode = 1) failed: rc = -2 [11943.394354] LustreError: 360579:0:(mgs_handler.c:1214:mgs_iocontrol()) MGS: setparam err: rc = -2 [11947.508557] LustreError: 359613:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11948.865794] LustreError: 359646:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [11948.877913] LustreError: 359646:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [11948.886220] LustreError: 359646:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 22, retries 0, failed: rc = -5 [11949.228534] Lustre: server umount lustre-MDT0000 complete [11960.130283] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [11962.671736] Key type lgssc unregistered [11962.955463] LNet: 361679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11962.962581] LNetError: 361679:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11962.994803] LNet: Removed LNI 192.168.201.112@tcp [11963.945522] Key type .llcrypt unregistered [11963.948040] Key type ._llcrypt unregistered [11982.264754] Key type ._llcrypt registered [11982.268283] Key type .llcrypt registered [11982.423913] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [11982.727397] Key type .llcrypt unregistered [11982.731417] Key type ._llcrypt unregistered [12000.503182] Key type ._llcrypt registered [12000.506618] Key type .llcrypt registered [12000.605577] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [12001.738958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12001.806026] alg: No test for adler32 (adler32-zlib) [12002.918728] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [12003.208688] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [12004.847163] Key type lgssc registered [12006.218293] Lustre: Echo OBD driver; http://www.lustre.org/ [12016.635554] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12016.657866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12018.384756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12023.546447] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12027.885986] LustreError: 363893:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12027.913940] LustreError: 363893:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12033.000268] LustreError: 363894:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12034.095131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12034.709533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12039.998673] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12046.939349] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12051.730943] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12057.429990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12057.695559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12059.751113] LustreError: 365770:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12062.511302] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12062.825570] LustreError: 365771:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12062.846948] LustreError: 365771:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12068.133381] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12068.332190] LustreError: 365770:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12068.358510] LustreError: 365770:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12078.565113] LustreError: 365773:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12078.582781] LustreError: 365773:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [12081.015796] loop1: detected capacity change from 0 to 204800000 [12081.209832] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12081.218993] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12081.229641] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12086.191244] loop1: detected capacity change from 0 to 204800000 [12091.187160] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12091.283848] LustreError: 363881:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs30955: New filesystem registration disabled. Use 'lctl set_param allow_register=unlimited' to allow new filesystem registrations: rc = -13 [12091.295812] Lustre: First target registration detected: opening a one-time 24-hour registration window. Use 'lctl set_param allow_register=0' to close it early, or 'lctl set_param -P allow_register=0' to lock down permanently. [12091.330382] Lustre: Setting parameter fs30955-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs30955-MDT0001 [12091.372390] Lustre: srv-fs30955-MDT0001: No data found on store. Initialize space. [12091.447308] Lustre: fs30955-MDT0001: new disk, initializing [12091.517654] Lustre: fs30955-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12091.537447] Lustre: srv-fs30955-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [12091.545080] Lustre: 367532:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs30955-MDT0001-mdtlov: can't create new llog: rc = -115 [12091.555673] LustreError: 367532:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs30955-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [12095.155391] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12095.379232] LustreError: 365770:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12095.389418] LustreError: 365770:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [12118.787436] loop2: detected capacity change from 0 to 204800000 [12119.029774] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12119.044100] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12119.055523] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12123.092440] loop2: detected capacity change from 0 to 204800000 [12127.725789] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12127.853665] Lustre: fs11571-OST0000: new disk, initializing [12127.856691] Lustre: srv-fs11571-OST0000: No data found on store. Initialize space. [12127.865131] Lustre: 369089:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12127.934503] Lustre: fs11571-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12129.773252] LustreError: 365770:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12129.788774] LustreError: 365770:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 19 previous similar messages [12133.038805] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12144.184134] Lustre: server umount fs11571-OST0000 complete [12145.126647] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12152.305342] Lustre: server umount fs30955-MDT0001 complete [12162.707608] Lustre: New target registration enabled indefinitely [12165.608880] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12165.619301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12165.625729] Lustre: Skipped 1 previous similar message [12165.634403] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12165.641367] LustreError: Skipped 1 previous similar message [12170.636865] Lustre: server umount lustre-OST0000 complete [12175.841086] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12175.848490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12175.852837] Lustre: Skipped 2 previous similar messages [12179.894467] Lustre: server umount lustre-MDT0000 complete [12182.756750] LustreError: 363874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789400089 with bad export cookie 5618176449780897619 [12182.763255] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12182.770611] LustreError: 363874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12183.061164] Lustre: server umount lustre-MDT0001 complete [12191.530921] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [12193.454327] Key type lgssc unregistered [12193.702117] LNet: 372133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12193.711054] LNetError: 372133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12193.738495] LNet: Removed LNI 192.168.201.112@tcp [12194.260951] Key type .llcrypt unregistered [12194.265492] Key type ._llcrypt unregistered [12220.820480] Key type ._llcrypt registered [12220.821801] Key type .llcrypt registered [12220.891854] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [12233.200651] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [12234.141412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12234.305633] alg: No test for adler32 (adler32-zlib) [12235.266410] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [12235.469525] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [12237.161339] Key type lgssc registered [12238.023672] Lustre: Echo OBD driver; http://www.lustre.org/ [12266.662673] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12266.683875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12267.973446] LustreError: 375967:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) lustre: New filesystem registration disabled. Use 'lctl set_param allow_register=unlimited' to allow new filesystem registrations: rc = -13 [12267.983467] Lustre: First target registration detected: opening a one-time 24-hour registration window. Use 'lctl set_param allow_register=0' to close it early, or 'lctl set_param -P allow_register=0' to lock down permanently. [12268.009225] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12268.122676] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12268.173289] Lustre: lustre-MDT0000: new disk, initializing [12268.255571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12268.271269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12271.912088] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12276.577106] LustreError: 376499:0:(llog.c:449:llog_init_handle()) llog type is not specified! [12277.865303] Lustre: New target registration enabled indefinitely [12284.013565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12284.133091] Lustre: 375965:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12284.190533] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12284.195121] Lustre: Skipped 1 previous similar message [12284.262722] Lustre: lustre-MDT0001: new disk, initializing [12284.360673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12284.406456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12284.423421] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12288.760902] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12297.199279] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12303.797922] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12311.142933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12311.506614] Lustre: lustre-OST0000: new disk, initializing [12311.512627] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12311.535171] Lustre: 378178:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12311.653914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12317.620456] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12321.334042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12321.345963] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12321.496310] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12329.438264] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12332.118285] Lustre: Mounted lustre-client - version 2.17.58_40_gbbf22b4 [12333.319350] Lustre: Unmounted lustre-client [12340.821512] LustreError: 379336:0:(llog.c:449:llog_init_handle()) llog type is not specified! [12347.364653] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12347.366288] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12347.392488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12350.209041] Lustre: server umount lustre-OST0000 complete [12356.064807] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12356.075869] LustreError: Skipped 1 previous similar message [12356.080500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12356.089438] Lustre: Skipped 1 previous similar message [12356.096094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12356.099082] Lustre: Skipped 1 previous similar message [12357.602638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12357.610633] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12359.510667] Lustre: server umount lustre-MDT0000 complete [12362.404461] LustreError: 377014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789400269 with bad export cookie 12193780119374479001 [12362.411340] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12362.412856] LustreError: 377014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12362.879967] Lustre: server umount lustre-MDT0001 complete [12371.372216] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [12373.451926] Key type lgssc unregistered [12391.199148] LNet: 1 local NIs in recovery (showing 1): 192.168.201.112@tcp [12392.096403] LNet: 380486:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12392.105220] LNetError: 380486:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12461.734346] LNet: Removed LNI 192.168.201.112@tcp [12462.554883] Key type .llcrypt unregistered [12462.557369] Key type ._llcrypt unregistered [12479.528866] Key type ._llcrypt registered [12479.531560] Key type .llcrypt registered [12479.634041] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [12497.223064] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [12498.779825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12498.965881] alg: No test for adler32 (adler32-zlib) [12500.126113] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [12500.434615] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [12502.175546] Key type lgssc registered [12504.015978] Lustre: Echo OBD driver; http://www.lustre.org/ [12536.175812] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 11:40:40 (1789400440) [12542.308448] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12542.355837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12543.621707] LustreError: 383706:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) lustre: New filesystem registration disabled. Use 'lctl set_param allow_register=unlimited' to allow new filesystem registrations: rc = -13 [12543.638190] Lustre: First target registration detected: opening a one-time 24-hour registration window. Use 'lctl set_param allow_register=0' to close it early, or 'lctl set_param -P allow_register=0' to lock down permanently. [12543.688422] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12543.733196] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12543.866803] Lustre: lustre-MDT0000: new disk, initializing [12543.968577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12543.982355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12548.403241] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12554.957286] LustreError: 384238:0:(llog.c:449:llog_init_handle()) llog type is not specified! [12557.346424] Lustre: New target registration enabled indefinitely [12567.739922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12567.916104] Lustre: 383703:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12567.971066] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12567.978097] Lustre: Skipped 1 previous similar message [12568.146756] Lustre: lustre-MDT0001: new disk, initializing [12568.340120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12568.450074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12568.470070] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12575.076565] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12584.237649] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12590.260424] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12598.411571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12598.919199] Lustre: lustre-OST0000: new disk, initializing [12598.930420] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12598.937851] Lustre: 385919:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12599.073113] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12605.938731] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12609.094463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12609.111321] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12609.275367] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12614.704703] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12627.290419] LustreError: 386824:0:(llog.c:449:llog_init_handle()) llog type is not specified! [12633.443225] Lustre: server umount lustre-OST0000 complete [12640.227424] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12640.230723] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12640.234674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12640.257709] Lustre: Skipped 1 previous similar message [12645.352398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12645.365634] Lustre: Skipped 2 previous similar messages [12647.499592] Lustre: server umount lustre-MDT0000 complete [12650.463485] Lustre: 381880:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789400541/real 1789400541] req@00000000578b76b3 x1876322321207552/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789400557 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12650.475391] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12650.772578] LustreError: 383695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789400557 with bad export cookie 11976208636804013706 [12650.773751] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12650.780611] LustreError: 383695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12651.092540] Lustre: server umount lustre-MDT0001 complete [12660.532980] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [12663.025226] Key type lgssc unregistered [12663.252564] LNet: 387977:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12663.255562] LNetError: 387977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12663.265053] LNet: Removed LNI 192.168.201.112@tcp [12664.005680] Key type .llcrypt unregistered [12664.010942] Key type ._llcrypt unregistered [12685.331441] Key type ._llcrypt registered [12685.332978] Key type .llcrypt registered [12685.420610] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [12686.529285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12686.608082] alg: No test for adler32 (adler32-zlib) [12687.866452] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [12688.167490] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [12689.832752] Key type lgssc registered [12691.037215] Lustre: Echo OBD driver; http://www.lustre.org/ [12699.507420] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12699.551264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12701.084206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12705.275370] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12709.534192] LustreError: 389718:0:(llog.c:449:llog_init_handle()) llog type is not specified! [12711.270257] Lustre: New target registration enabled indefinitely [12711.406491] LustreError: 389247:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12711.427089] LustreError: 389247:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12716.518700] LustreError: 389248:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12719.116136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12719.470207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12724.277846] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12731.614372] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12736.201770] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12742.569672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12743.112536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12749.046947] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12757.186336] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12767.430907] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12769.116819] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12775.339824] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.12@tcp (stopping) [12775.905410] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12775.915239] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12775.935312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12777.952567] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12777.953512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12777.981271] Lustre: Skipped 1 previous similar message [12778.002336] Lustre: Skipped 2 previous similar messages [12779.884548] Lustre: server umount lustre-MDT0000 complete [12780.444619] LustreError: 390495:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12783.074182] LustreError: 389248:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12783.091251] LustreError: 389248:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12784.320132] LustreError: 389230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789400690 with bad export cookie 1813848306914943120 [12784.356273] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12785.575557] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.12@tcp (stopping) [12788.199591] LustreError: 389244:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12788.219120] LustreError: 389244:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12788.224362] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12790.979784] Lustre: server umount lustre-MDT0001 complete [12796.962968] Lustre: server umount lustre-OST0000 complete [12804.580525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12809.353253] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12813.074409] Lustre: 393719:0:(mgs_llog.c:1811:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.112@tcp [12815.462135] Lustre: server umount MGS complete [12823.326613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12823.779296] LustreError: 394292:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12823.794964] LustreError: 394292:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12823.853084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12827.985343] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12837.139901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12837.587611] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12841.760381] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12849.458187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12849.725854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12854.769855] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12861.287448] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12862.535835] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12866.756571] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12874.354542] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 6 sec [12876.899861] LustreError: 396770:0:(llog.c:449:llog_init_handle()) llog type is not specified! [12883.425124] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12883.432425] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12883.443181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12883.453412] Lustre: Skipped 2 previous similar messages [12884.454884] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12886.124644] Lustre: server umount lustre-OST0000 complete [12889.568789] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12889.586631] Lustre: Skipped 1 previous similar message [12894.178491] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12894.183804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12894.186293] Lustre: Skipped 3 previous similar messages [12895.483231] Lustre: server umount lustre-MDT0000 complete [12898.868312] LustreError: 394273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789400805 with bad export cookie 1813848306914943960 [12898.878736] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12898.881377] LustreError: 394273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [12899.138808] Lustre: server umount lustre-MDT0001 complete [12908.187394] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [12910.813756] Key type lgssc unregistered [12911.074907] LNet: 397923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12911.084479] LNetError: 397923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12911.106483] LNet: Removed LNI 192.168.201.112@tcp [12911.746811] Key type .llcrypt unregistered [12911.749052] Key type ._llcrypt unregistered [12928.645248] Key type ._llcrypt registered [12928.646979] Key type .llcrypt registered [12928.772608] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [12944.371655] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [12945.209889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12945.367538] alg: No test for adler32 (adler32-zlib) [12946.440729] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [12946.671529] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [12948.391194] Key type lgssc registered [12949.535574] Lustre: Echo OBD driver; http://www.lustre.org/ [12978.821112] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12978.857744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12980.007853] LustreError: 401084:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) lustre: New filesystem registration disabled. Use 'lctl set_param allow_register=unlimited' to allow new filesystem registrations: rc = -13 [12980.032484] Lustre: First target registration detected: opening a one-time 24-hour registration window. Use 'lctl set_param allow_register=0' to close it early, or 'lctl set_param -P allow_register=0' to lock down permanently. [12980.078490] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12980.097458] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12980.148456] Lustre: lustre-MDT0000: new disk, initializing [12980.224783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12980.242567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12983.525466] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12987.745286] LustreError: 401616:0:(llog.c:449:llog_init_handle()) llog type is not specified! [12988.908892] Lustre: New target registration enabled indefinitely [12993.654384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12993.740779] Lustre: 401081:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12993.782420] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12993.786859] Lustre: Skipped 1 previous similar message [12993.831282] Lustre: lustre-MDT0001: new disk, initializing [12993.901718] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12993.929895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12993.944122] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12996.748356] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [13002.755938] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13006.128811] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13010.062813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13010.201981] Lustre: lustre-OST0000: new disk, initializing [13010.204321] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13010.207764] Lustre: 403294:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13010.274977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13012.050765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13012.054916] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13012.125148] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13013.696239] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [13020.096587] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13023.455363] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13023.611781] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13026.458699] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13026.618257] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13032.931282] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13032.931397] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13032.937548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13032.942705] Lustre: Skipped 1 previous similar message [13038.048802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13038.055102] Lustre: Skipped 2 previous similar messages [13042.146046] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13042.268388] Lustre: server umount lustre-OST0000 complete [13045.221091] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13045.232310] LustreError: Skipped 1 previous similar message [13045.234344] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13045.239914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13048.289128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13048.289577] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13050.774930] Lustre: server umount lustre-MDT0000 complete [13053.704750] LustreError: 401074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789400960 with bad export cookie 4077871843025524950 [13053.716260] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13053.719498] LustreError: 401074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13053.965562] Lustre: server umount lustre-MDT0001 complete [13059.544656] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 11:49:25 (1789400965) [13069.916491] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [13076.496883] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [13102.298995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13102.485467] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13102.519126] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13102.614829] Lustre: lustre-MDT0000: new disk, initializing [13102.692412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13102.703601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13106.195332] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [13115.750449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13115.846241] Lustre: 408046:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13115.929435] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13115.937394] Lustre: Skipped 1 previous similar message [13116.018899] Lustre: lustre-MDT0001: new disk, initializing [13116.091682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13116.117066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13116.128549] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13121.198687] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [13128.500943] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13132.461705] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13137.580618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13137.759235] Lustre: lustre-OST0000: new disk, initializing [13137.762518] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13137.768515] Lustre: 410063:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13137.845760] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13141.971939] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [13145.112644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13145.118684] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13145.161170] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13148.489444] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13166.851747] LustreError: 411209:0:(llog.c:449:llog_init_handle()) llog type is not specified! [13170.659063] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13170.668425] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13170.684036] LustreError: Skipped 1 previous similar message [13170.694982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13170.702039] Lustre: Skipped 2 previous similar messages [13175.489900] Lustre: server umount lustre-OST0000 complete [13178.152288] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13178.157905] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13178.164952] Lustre: Skipped 1 previous similar message [13180.897701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13180.906972] Lustre: Skipped 2 previous similar messages [13184.335583] Lustre: server umount lustre-MDT0000 complete [13186.020828] LustreError: 408058:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13186.038951] LustreError: 408058:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [13186.794982] LustreError: 408039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789401093 with bad export cookie 4077871843025525832 [13186.796374] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13186.799736] LustreError: 408039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13186.968640] Lustre: server umount lustre-MDT0001 complete [13193.528798] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [13195.495497] Key type lgssc unregistered [13195.696654] LNet: 412356:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13195.700376] LNetError: 412356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13195.721519] LNet: Removed LNI 192.168.201.112@tcp [13196.361659] Key type .llcrypt unregistered [13196.363873] Key type ._llcrypt unregistered [13273.675851] Key type ._llcrypt registered [13273.677773] Key type .llcrypt registered [13273.739327] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [13273.920299] Key type .llcrypt unregistered [13273.921978] Key type ._llcrypt unregistered [13330.226485] Key type ._llcrypt registered [13330.228795] Key type .llcrypt registered [13330.320795] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [13330.577575] Key type .llcrypt unregistered [13330.580164] Key type ._llcrypt unregistered [13356.234881] Key type ._llcrypt registered [13356.237304] Key type .llcrypt registered [13356.328935] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [13356.613393] Key type .llcrypt unregistered [13356.615171] Key type ._llcrypt unregistered [13383.835658] Key type ._llcrypt registered [13383.838265] Key type .llcrypt registered [13383.918698] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [13384.230252] Key type .llcrypt unregistered [13384.231438] Key type ._llcrypt unregistered [13471.008222] Key type ._llcrypt registered [13471.009649] Key type .llcrypt registered [13471.130938] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [13471.470403] Key type .llcrypt unregistered [13471.472040] Key type ._llcrypt unregistered [13501.116766] Key type ._llcrypt registered [13501.118536] Key type .llcrypt registered [13501.192683] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [13501.407412] Key type .llcrypt unregistered [13501.408709] Key type ._llcrypt unregistered [13536.505313] Key type ._llcrypt registered [13536.509800] Key type .llcrypt registered [13536.614621] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [13536.859409] Key type .llcrypt unregistered [13536.861106] Key type ._llcrypt unregistered [13568.371856] Key type ._llcrypt registered [13568.373804] Key type .llcrypt registered [13568.459869] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [13568.748863] Key type .llcrypt unregistered [13568.750627] Key type ._llcrypt unregistered [13598.474345] Key type ._llcrypt registered [13598.476485] Key type .llcrypt registered [13598.615278] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [13599.036600] Key type .llcrypt unregistered [13599.042583] Key type ._llcrypt unregistered [13628.508543] Key type ._llcrypt registered [13628.516845] Key type .llcrypt registered [13628.781102] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [13629.095529] Key type .llcrypt unregistered [13629.098031] Key type ._llcrypt unregistered [13710.243389] Key type ._llcrypt registered [13710.244748] Key type .llcrypt registered [13710.443128] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [13710.876712] Key type .llcrypt unregistered [13710.878776] Key type ._llcrypt unregistered [13751.685848] Key type ._llcrypt registered [13751.688416] Key type .llcrypt registered [13751.798421] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [13752.032282] Key type .llcrypt unregistered [13752.034791] Key type ._llcrypt unregistered [13788.152489] Key type ._llcrypt registered [13788.153874] Key type .llcrypt registered [13788.289700] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [13788.599540] Key type .llcrypt unregistered [13788.603506] Key type ._llcrypt unregistered [13840.519382] Key type ._llcrypt registered [13840.520539] Key type .llcrypt registered [13840.682290] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [13858.834956] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [13860.539900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13860.662291] alg: No test for adler32 (adler32-zlib) [13861.900407] Lustre: Lustre: Build Version: 2.17.58_40_gbbf22b4 [13862.281192] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [13864.040372] Key type lgssc registered [13865.346339] Lustre: Echo OBD driver; http://www.lustre.org/ [13907.324372] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13907.367245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13908.650098] LustreError: 434444:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) tst250fs: New filesystem registration disabled. Use 'lctl set_param allow_register=unlimited' to allow new filesystem registrations: rc = -13 [13908.658338] Lustre: First target registration detected: opening a one-time 24-hour registration window. Use 'lctl set_param allow_register=0' to close it early, or 'lctl set_param -P allow_register=0' to lock down permanently. [13908.671456] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13908.689596] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13908.740178] Lustre: tst250fs-MDT0000: new disk, initializing [13908.796553] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13908.808134] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13913.281694] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [13919.034880] LustreError: 434976:0:(llog.c:449:llog_init_handle()) llog type is not specified! [13921.171373] Lustre: New target registration enabled indefinitely [13928.828420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13928.959372] Lustre: 434443:0:(mgs_llog.c:1453:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13929.002207] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13929.006312] Lustre: Skipped 1 previous similar message [13929.086835] Lustre: tst250fs-MDT0001: new disk, initializing [13929.189075] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13929.221179] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13929.233140] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13933.539767] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [13938.996213] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13946.581881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13946.968746] Lustre: tst250fs-OST0000: new disk, initializing [13946.973350] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13946.982538] Lustre: 436316:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13947.117274] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13947.514526] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13947.530695] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13947.666884] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13955.371431] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [13969.003499] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13969.252495] Lustre: tst250fs-OST0001: new disk, initializing [13969.256354] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13969.267509] Lustre: 437190:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13969.382092] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13976.388394] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [13977.777155] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13977.785805] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13977.872997] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13982.207117] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13984.574357] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13984.578639] Lustre: Skipped 2 previous similar messages [14003.682127] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14003.702511] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14003.716207] Lustre: Skipped 1 previous similar message [14003.726667] Lustre: Skipped 2 previous similar messages [14008.808686] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14008.833515] Lustre: Skipped 3 previous similar messages [14009.504710] Lustre: server umount tst250fs-MDT0000 complete [14013.341924] LustreError: 434435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789401919 with bad export cookie 13005759230472556646 [14013.345583] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14013.361084] LustreError: 434435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14013.956902] Lustre: server umount tst250fs-MDT0001 complete [14027.655861] Lustre: server umount tst250fs-OST0000 complete [14030.303365] Lustre: 432661:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789401920/real 1789401920] req@ffff8c6e5b089500 x1876323748351488/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789401936 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14030.338338] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14030.370375] Lustre: Skipped 2 previous similar messages [14031.579249] Lustre: server umount tst250fs-OST0001 complete [14051.039939] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [14058.706871] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [14088.014395] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 12:06:32 (1789401992) [14089.936898] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14106.677772] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [14120.588228] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [14161.956381] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13778 sec ========= 12:07:47 (1789402067) [14163.750697] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:07:48 (1789402068) === [14167.813781] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:07:52 (1789402072) === [14211.718729] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [14214.623664] Key type lgssc unregistered [14215.036627] LNet: 446323:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14215.040858] LNetError: 446323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14215.052640] LNet: Removed LNI 192.168.201.112@tcp [14215.731195] Key type .llcrypt unregistered [14215.733641] Key type ._llcrypt unregistered