[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 415843378 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002224] x2apic enabled [ 0.003004] Switched APIC routing to physical x2apic. [ 0.003892] kvm-guest: setup PV IPIs [ 0.005823] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006013] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007006] pid_max: default: 32768 minimum: 301 [ 0.008109] LSM: Security Framework initializing [ 0.009024] Yama: becoming mindful. [ 0.010019] SELinux: Initializing. [ 0.010652] *** VALIDATE selinux *** [ 0.017120] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.021042] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.022102] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.023075] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024066] *** VALIDATE tmpfs *** [ 0.025318] *** VALIDATE proc *** [ 0.026148] *** VALIDATE cgroup *** [ 0.027004] *** VALIDATE cgroup2 *** [ 0.028172] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.029100] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.030002] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.031016] Spectre V2 : User space: Vulnerable [ 0.032004] Speculative Store Bypass: Vulnerable [ 0.035287] debug: unmapping init [mem 0xffffffffb2659000-0xffffffffb2660fff] [ 0.037176] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.038590] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.039015] ... version: 2 [ 0.039951] ... bit width: 48 [ 0.040008] ... generic registers: 4 [ 0.040943] ... value mask: 0000ffffffffffff [ 0.041009] ... max period: 00007fffffffffff [ 0.042007] ... fixed-purpose events: 3 [ 0.043006] ... event mask: 000000070000000f [ 0.044249] rcu: Hierarchical SRCU implementation. [ 0.046334] smp: Bringing up secondary CPUs ... [ 0.047450] x86: Booting SMP configuration: [ 0.048015] .... node #0, CPUs: #1 #2 #3 [ 0.050757] smp: Brought up 1 node, 4 CPUs [ 0.052014] smpboot: Max logical packages: 1 [ 0.053014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.260846] node 0 deferred pages initialised in 206ms [ 0.263090] devtmpfs: initialized [ 0.264201] x86/mm: Memory block size: 128MB [ 0.266542] gcov: version magic: 0x41383552 [ 0.267558] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.268054] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.269183] pinctrl core: initialized pinctrl subsystem [ 0.270143] [ 0.270527] ************************************************************* [ 0.271007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.272006] ** ** [ 0.273013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.274006] ** ** [ 0.275007] ** This means that this kernel is built to expose internal ** [ 0.276006] ** IOMMU data structures, which may compromise security on ** [ 0.277006] ** your system. ** [ 0.278006] ** ** [ 0.279006] ** If you see this message and you are not debugging the ** [ 0.280006] ** kernel, report this immediately to your vendor! ** [ 0.281011] ** ** [ 0.282006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.283006] ************************************************************* [ 0.284519] NET: Registered protocol family 16 [ 0.285406] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.286039] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.287033] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.288313] cpuidle: using governor menu [ 0.289515] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.291428] PCI: Using configuration type 1 for base access [ 0.292093] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.299159] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.300011] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.301168] cryptd: max_cpu_qlen set to 1000 [ 0.303776] ACPI: Added _OSI(Module Device) [ 0.305011] ACPI: Added _OSI(Processor Device) [ 0.306009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.308008] ACPI: Added _OSI(Processor Aggregator Device) [ 0.312124] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.318226] ACPI: Interpreter enabled [ 0.319042] ACPI: PM: (supports S0 S3 S4 S5) [ 0.319948] ACPI: Using IOAPIC for interrupt routing [ 0.321087] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.323250] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.331083] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.333022] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.335013] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.336044] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.339269] acpiphp: Slot [2] registered [ 0.340097] acpiphp: Slot [5] registered [ 0.340767] acpiphp: Slot [6] registered [ 0.341059] acpiphp: Slot [7] registered [ 0.341677] acpiphp: Slot [8] registered [ 0.343054] acpiphp: Slot [9] registered [ 0.343965] acpiphp: Slot [10] registered [ 0.344073] acpiphp: Slot [3] registered [ 0.344943] acpiphp: Slot [4] registered [ 0.346053] acpiphp: Slot [11] registered [ 0.346634] acpiphp: Slot [12] registered [ 0.347043] acpiphp: Slot [13] registered [ 0.347923] acpiphp: Slot [14] registered [ 0.349065] acpiphp: Slot [15] registered [ 0.350036] acpiphp: Slot [16] registered [ 0.350985] acpiphp: Slot [17] registered [ 0.352057] acpiphp: Slot [18] registered [ 0.352878] acpiphp: Slot [19] registered [ 0.353037] acpiphp: Slot [20] registered [ 0.353583] acpiphp: Slot [21] registered [ 0.354039] acpiphp: Slot [22] registered [ 0.355018] acpiphp: Slot [23] registered [ 0.355827] acpiphp: Slot [24] registered [ 0.357072] acpiphp: Slot [25] registered [ 0.357918] acpiphp: Slot [26] registered [ 0.358052] acpiphp: Slot [27] registered [ 0.358850] acpiphp: Slot [28] registered [ 0.360059] acpiphp: Slot [29] registered [ 0.360707] acpiphp: Slot [30] registered [ 0.361056] acpiphp: Slot [31] registered [ 0.361849] PCI host bridge to bus 0000:00 [ 0.362013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.362948] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.364010] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.365009] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.367009] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.368009] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.369163] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.370741] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.372817] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.381014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.386006] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.388013] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.389009] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.390008] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.392546] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.393585] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.394017] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.396528] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.400012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.410012] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.414009] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.418016] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.422021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.426010] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.440017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.448640] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.455025] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.461024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.478025] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.488255] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.497017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.503017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.517019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.531365] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.535013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.545022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.554019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.563471] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.568012] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.574020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.587020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.597501] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.603016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.608019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.625019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.635454] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.637307] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.638276] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.640212] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.642144] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.645585] iommu: Default domain type: Passthrough [ 0.646481] SCSI subsystem initialized [ 0.647038] ACPI: bus type USB registered [ 0.647911] usbcore: registered new interface driver usbfs [ 0.649038] usbcore: registered new interface driver hub [ 0.650043] usbcore: registered new device driver usb [ 0.651051] pps_core: LinuxPPS API ver. 1 registered [ 0.651893] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.654045] PTP clock support registered [ 0.655144] EDAC MC: Ver: 3.0.0 [ 0.656436] PCI: Using ACPI for IRQ routing [ 0.657788] NetLabel: Initializing [ 0.658007] NetLabel: domain hash size = 128 [ 0.658972] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.659081] NetLabel: unlabeled traffic allowed by default [ 0.660052] vgaarb: loaded [ 0.660462] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.662012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.667007] clocksource: Switched to clocksource kvm-clock [ 0.751993] VFS: Disk quotas dquot_6.6.0 [ 0.753448] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.755747] *** VALIDATE ramfs *** [ 0.756974] *** VALIDATE hugetlbfs *** [ 0.758374] pnp: PnP ACPI init [ 0.760657] pnp: PnP ACPI: found 6 devices [ 0.775519] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.778744] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.781237] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.783265] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.786024] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.787970] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.789926] NET: Registered protocol family 2 [ 0.791712] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.795956] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.799824] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.804111] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.806645] TCP: Hash tables configured (established 65536 bind 65536) [ 0.808668] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.811093] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.813046] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.815239] NET: Registered protocol family 1 [ 0.817441] RPC: Registered named UNIX socket transport module. [ 0.818992] RPC: Registered udp transport module. [ 0.820240] RPC: Registered tcp transport module. [ 0.821233] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.822598] NET: Registered protocol family 44 [ 0.823802] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.825448] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.827067] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.828735] PCI: CLS 0 bytes, default 64 [ 0.830589] Unpacking initramfs... [ 2.133248] debug: unmapping init [mem 0xffff96b93cc54000-0xffff96b93ffbffff] [ 2.136779] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.138307] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.140350] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.567287] Initialise system trusted keyrings [ 2.568685] Key type blacklist registered [ 2.570032] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.577808] zbud: loaded [ 2.580190] *** VALIDATE nfs *** [ 2.581113] *** VALIDATE nfs4 *** [ 2.582829] pstore: using deflate compression [ 2.585572] Platform Keyring initialized [ 2.653134] NET: Registered protocol family 38 [ 2.654334] Key type asymmetric registered [ 2.655355] Asymmetric key parser 'x509' registered [ 2.656561] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.658604] io scheduler mq-deadline registered [ 2.659863] io scheduler kyber registered [ 2.662147] io scheduler bfq registered [ 2.663370] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.665314] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.667246] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.669116] ACPI: Power Button [PWRF] [ 2.673504] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.678927] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.699483] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.709065] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.737594] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.763181] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.789638] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.794472] Non-volatile memory driver v1.3 [ 2.796180] Linux agpgart interface v0.103 [ 2.822516] virtio_blk virtio1: [vda] 146728 512-byte logical blocks (75.1 MB/71.6 MiB) [ 2.825035] vda: detected capacity change from 0 to 75124736 [ 2.843639] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.845552] vdb: detected capacity change from 0 to 1073741824 [ 2.856583] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.858985] vdc: detected capacity change from 0 to 2621440000 [ 2.874262] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.877221] vdd: detected capacity change from 0 to 2621440000 [ 2.895516] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.897734] vde: detected capacity change from 0 to 4294967296 [ 2.910318] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.912645] vdf: detected capacity change from 0 to 4294967296 [ 2.919235] libphy: Fixed MDIO Bus: probed [ 2.926367] usbcore: registered new interface driver usbserial_generic [ 2.929141] usbserial: USB Serial support registered for generic [ 2.931787] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.936437] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.938476] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.941942] mousedev: PS/2 mouse device common for all mice [ 2.944523] rtc_cmos 00:05: RTC can wake from S4 [ 2.947304] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.951506] rtc_cmos 00:05: registered as rtc0 [ 2.954568] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.955835] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.960323] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.962203] intel_pstate: CPU model not supported [ 2.967813] hid: raw HID events driver (C) Jiri Kosina [ 2.969141] usbcore: registered new interface driver usbhid [ 2.970576] usbhid: USB HID core driver [ 2.972107] drop_monitor: Initializing network drop monitor service [ 2.973843] Initializing XFRM netlink socket [ 2.975335] NET: Registered protocol family 10 [ 2.977848] Segment Routing with IPv6 [ 2.978796] NET: Registered protocol family 17 [ 2.980130] mpls_gso: MPLS GSO support [ 2.989928] RAS: Correctable Errors collector initialized. [ 2.992040] AVX version of gcm_enc/dec engaged. [ 2.993722] AES CTR mode by8 optimization enabled [ 3.062616] sched_clock: Marking stable (3062597341, 0)->(3719243224, -656645883) [ 3.066302] registered taskstats version 1 [ 3.067875] Loading compiled-in X.509 certificates [ 3.069614] zswap: loaded using pool lzo/zbud [ 3.090204] Key type big_key registered [ 3.099240] Key type encrypted registered [ 3.100340] ima: No TPM chip found, activating TPM-bypass! [ 3.102249] ima: Allocated hash algorithm: sha1 [ 3.104059] ima: No architecture policies found [ 3.105831] evm: Initialising EVM extended attributes: [ 3.107495] evm: security.selinux [ 3.108552] evm: security.ima [ 3.109408] evm: security.capability [ 3.110592] evm: HMAC attrs: 0x1 [ 3.112869] rtc_cmos 00:05: setting system clock to 2026-09-08 12:10:43 UTC (1788869443) [ 3.119630] debug: unmapping init [mem 0xffffffffb3603000-0xffffffffb37fffff] [ 3.123540] debug: unmapping init [mem 0xffffffffb2382000-0xffffffffb2658fff] [ 3.132066] Write protecting the kernel read-only data: 28672k [ 3.137626] debug: unmapping init [mem 0xffffffffb0a03000-0xffffffffb0bfffff] [ 3.140412] debug: unmapping init [mem 0xffffffffb1314000-0xffffffffb13fffff] [ 3.167252] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.174311] systemd[1]: Detected virtualization kvm. [ 3.175523] systemd[1]: Detected architecture x86-64. [ 3.176549] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.205475] systemd[1]: No hostname configured. [ 3.207939] systemd[1]: Set hostname to . [ 3.209690] random: systemd: uninitialized urandom read (16 bytes read) [ 3.211551] systemd[1]: Initializing machine ID from random generator. [ 3.333221] random: systemd: uninitialized urandom read (16 bytes read) [ 3.335437] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.340730] random: systemd: uninitialized urandom read (16 bytes read) [ 3.342878] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.348326] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.951434] device-mapper: uevent: version 1.0.3 [ 3.953355] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.591754] random: fast init done [ 4.593758] virtio_net virtio0 ens2: renamed from eth0 [ 4.650936] scsi host0: ata_piix [ 4.671194] scsi host1: ata_piix [ 4.672269] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.673638] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.975235] dracut-initqueue[597]: RTNETLINK answers: File exists [ 9.620961] random: crng init done [ 9.622064] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.860218] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.798450] printk: systemd: 26 output lines suppressed due to ratelimiting [ 10.951258] SELinux: Disabled at runtime. [ 11.016123] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.021751] systemd[1]: Detected virtualization kvm. [ 11.023020] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.379249] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.381385] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.385419] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.390289] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.392628] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.398177] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.401811] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... [ 11.453718] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... Starting udev Coldplug all Devices... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 11.783521] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.058097] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.074670] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.166425] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.201645] EDAC sbridge: Ver: 1.1.2 [ 13.360952] Key type dns_resolver registered [ 13.698626] NFS: Registering the id_resolver key type [ 13.701292] Key type id_resolver registered [ 13.703045] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg216-server login: [ 34.731669] libcfs: loading out-of-tree module taints kernel. [ 34.746858] Key type ._llcrypt registered [ 34.747710] Key type .llcrypt registered [ 34.779937] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 42.018521] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 42.577920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 42.584804] alg: No test for adler32 (adler32-zlib) [ 43.574732] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 43.897228] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 45.503163] Key type lgssc registered [ 46.261987] Lustre: Echo OBD driver; http://www.lustre.org/ [ 53.181560] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 66.853734] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 71.264566] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 71.270977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 72.352993] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 72.363814] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 72.399920] Lustre: lustre-MDT0000: new disk, initializing [ 72.425377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 72.432490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 73.775544] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 79.418269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 79.457791] Lustre: 6483:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 79.472743] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 79.475384] Lustre: Skipped 1 previous similar message [ 79.520372] Lustre: lustre-MDT0001: new disk, initializing [ 79.546798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 79.559052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 79.566592] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 81.304888] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 84.655285] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 91.783229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 91.974479] Lustre: lustre-OST0000: new disk, initializing [ 91.977659] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 91.981706] Lustre: 8420:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 92.032521] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 95.110925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 95.115937] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 95.176084] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 97.204577] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 99.951085] hrtimer: interrupt took 4172866 ns [ 109.176737] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 109.301847] Lustre: lustre-OST0001: new disk, initializing [ 109.304625] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 109.318953] Lustre: 9491:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 109.398738] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 114.844177] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 117.773850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 117.787386] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 117.856889] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 124.881256] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 137.469034] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 151.203847] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing check_logdir /tmp/testlogs/ [ 162.077052] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing yml_node [ 170.838761] Lustre: DEBUG MARKER: Client: 2.17.57.113 [ 175.690844] Lustre: DEBUG MARKER: MDS: 2.17.57.113 [ 180.804844] Lustre: DEBUG MARKER: OSS: 2.17.57.113 [ 183.858362] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 8 08:13:40 EDT 2026 [ 205.040865] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 206.884913] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 223.202119] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 223.218701] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 223.235251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 225.249278] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 225.249445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 228.524356] Lustre: server umount lustre-MDT0000 complete [ 231.454687] LustreError: 6476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869671 with bad export cookie 12444550015083217728 [ 231.464071] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 231.470673] LustreError: 6476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 235.493553] LustreError: 6488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 235.495822] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 235.501653] LustreError: 6488:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 235.502648] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 235.509791] Lustre: Skipped 3 previous similar messages [ 235.528759] Lustre: Skipped 4 previous similar messages [ 236.516486] LustreError: 6517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 237.790563] Lustre: server umount lustre-MDT0001 complete [ 241.015662] Lustre: server umount lustre-OST0000 complete [ 244.182227] Lustre: server umount lustre-OST0001 complete [ 250.585420] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 258.199618] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 295.553042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 295.936408] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 295.995740] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 296.152240] Lustre: lustre-MDT0000: new disk, initializing [ 296.255306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 296.293674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 300.561682] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 311.163336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 311.313148] Lustre: 15044:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 311.360373] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 311.363498] Lustre: Skipped 1 previous similar message [ 311.463282] Lustre: lustre-MDT0001: new disk, initializing [ 311.542530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 311.599287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 311.604427] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 315.582432] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 323.764309] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 328.750600] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 334.578226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 334.802188] Lustre: lustre-OST0000: new disk, initializing [ 334.805230] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 334.812739] Lustre: 17010:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 334.875809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 336.691275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 336.713533] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 336.873911] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 340.438377] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 354.123264] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 361.318500] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 361.765758] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 367.827790] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 368.130784] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 372.712288] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 372.715213] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 372.724814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 372.730812] Lustre: Skipped 1 previous similar message [ 372.747569] Lustre: Skipped 3 previous similar messages [ 377.851230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 377.861024] Lustre: Skipped 1 previous similar message [ 384.992605] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 385.176363] Lustre: server umount lustre-OST0000 complete [ 393.189588] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 393.194248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 393.208527] Lustre: Skipped 1 previous similar message [ 393.212029] Lustre: Skipped 2 previous similar messages [ 395.684384] Lustre: server umount lustre-MDT0000 complete [ 398.308875] LustreError: 15958:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 398.323962] LustreError: 15958:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 399.391164] LustreError: 15036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869839 with bad export cookie 12444550015083219261 [ 399.399756] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 399.844968] Lustre: server umount lustre-MDT0001 complete [ 406.936783] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 08:17:24 (1788869844) [ 427.318960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 428.035365] LustreError: 19892:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 428.105644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 433.121201] LustreError: 19893:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 433.456378] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 442.493888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 447.708176] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 456.412796] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 462.557815] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 465.216226] Lustre: Failing over lustre-MDT0000 [ 465.581803] Lustre: server umount lustre-MDT0000 complete [ 468.326137] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 468.352139] LustreError: 19888:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 468.380761] LustreError: 19888:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 483.813115] Lustre: 3618:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788869908/real 1788869908] req@ffff96b9bf84b800 x1875765608225536/t0(0) o400->MGC192.168.202.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788869924 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 483.849740] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 485.944881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 487.969902] LustreError: 19887:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 487.991272] LustreError: 19887:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 495.075410] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xacb3ec4dfe9e0ec3 [ 495.083095] Lustre: MGC192.168.202.116@tcp: Connection restored to 0@lo (at 0@lo) [ 495.362953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 495.367603] Lustre: Skipped 1 previous similar message [ 500.413409] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 500.716944] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 500.731198] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 500.749574] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 512.017219] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 518.122740] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 518.134158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 518.140700] Lustre: Skipped 2 previous similar messages [ 518.172937] Lustre: Skipped 2 previous similar messages [ 523.985728] Lustre: server umount lustre-MDT0000 complete [ 527.575825] LustreError: 19874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869967 with bad export cookie 12444550015083220675 [ 527.580373] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 527.582747] LustreError: 19874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 527.818779] Lustre: server umount lustre-MDT0001 complete [ 536.491266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 536.856702] LustreError: 23229:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 536.878285] LustreError: 23229:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 542.132737] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 552.629270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 558.094227] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 567.268516] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 573.724513] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 576.662918] Lustre: Failing over lustre-MDT0000 [ 577.097425] Lustre: server umount lustre-MDT0000 complete [ 577.698468] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 594.400896] Lustre: 3619:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788870018/real 1788870018] req@ffff96b9bf81d180 x1875765608256896/t0(0) o400->MGC192.168.202.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788870034 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 594.437209] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 598.216269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 603.690396] LustreError: 23230:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 603.735168] LustreError: 23230:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [ 604.120616] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 609.262193] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 609.277760] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 609.295554] Lustre: Skipped 1 previous similar message [ 609.337577] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 609.717736] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 621.639726] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 627.684835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 627.705920] Lustre: Skipped 3 previous similar messages [ 633.519439] Lustre: server umount lustre-MDT0000 complete [ 638.282076] LustreError: 24801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870078 with bad export cookie 12444550015083221620 [ 638.285147] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 638.305298] LustreError: 24801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 638.825130] Lustre: server umount lustre-MDT0001 complete [ 654.110856] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 08:21:30 (1788870090) [ 677.265897] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 690.691834] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 746.420641] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 759.532141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 759.789387] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 759.809165] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 759.918704] Lustre: lustre-MDT0000: new disk, initializing [ 760.129539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 760.141564] Lustre: Skipped 3 previous similar messages [ 760.189136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 765.267253] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 777.498239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 777.670131] Lustre: 29631:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 777.900463] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 783.143350] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 795.286961] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 801.557931] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 809.804627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 810.119050] Lustre: 31601:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 812.156650] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 812.205498] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 812.314321] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 812.334053] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 817.595732] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 828.357522] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 834.565049] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 858.594171] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 858.600981] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 858.623340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 858.623349] Lustre: Skipped 2 previous similar messages [ 858.633460] Lustre: Skipped 4 previous similar messages [ 858.649471] LustreError: Skipped 1 previous similar message [ 861.480836] Lustre: server umount lustre-OST0000 complete [ 865.248538] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 871.972412] Lustre: server umount lustre-MDT0000 complete [ 873.952465] LustreError: 29643:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 873.965517] LustreError: 29643:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 876.229157] LustreError: 29623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870316 with bad export cookie 12444550015083222026 [ 876.239796] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 876.244064] LustreError: 29623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 876.868731] Lustre: server umount lustre-MDT0001 complete [ 894.129100] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 897.235791] Key type lgssc unregistered [ 897.491773] LNet: 33577:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 897.507980] LNetError: 33577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 897.528979] LNet: Removed LNI 192.168.202.116@tcp [ 898.330736] Key type .llcrypt unregistered [ 898.333699] Key type ._llcrypt unregistered [ 927.404603] Key type ._llcrypt registered [ 927.407833] Key type .llcrypt registered [ 927.571497] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 928.953050] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 929.130137] alg: No test for adler32 (adler32-zlib) [ 930.271160] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 930.476695] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 932.135205] Key type lgssc registered [ 933.358730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 941.761422] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 941.789562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 943.344406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 948.405766] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 952.816527] LustreError: 34990:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 952.841129] LustreError: 34990:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 957.923191] LustreError: 34991:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 959.850438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 960.309190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 966.407232] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 976.276924] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 984.201587] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 992.264894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 992.819655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1000.740912] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1005.129269] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1010.422597] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1013.330821] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1033.001843] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 08:27:50 (1788870470) [ 1035.608920] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1037.863125] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1046.106390] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 08:28:03 (1788870483) [ 1057.809871] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 08:28:15 (1788870495) [ 1059.238649] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1070.168920] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1070.173784] Lustre: Skipped 2 previous similar messages [ 1072.475951] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 08:28:29 (1788870509) [ 1074.846110] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1074.850678] Lustre: Skipped 2 previous similar messages [ 1084.752717] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1084.758756] Lustre: Skipped 4 previous similar messages [ 1101.231238] Lustre: Disabling parameter general.osc.*.max_pages_per_rpc= in log params [ 1101.237716] Lustre: Skipped 5 previous similar messages [ 1103.663260] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 08:29:01 (1788870541) [ 1106.801193] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1111.146531] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1120.881879] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 08:29:18 (1788870558) [ 1136.561599] Lustre: Setting parameter general.jobid_name=jobid_name=jobid_name=%e.root in log params [ 1136.567653] Lustre: Skipped 4 previous similar messages [ 1139.252551] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 08:29:36 (1788870576) [ 1156.580389] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1156.582727] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1156.605800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1156.636501] Lustre: Skipped 1 previous similar message [ 1161.704463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1161.716587] Lustre: Skipped 2 previous similar messages [ 1162.325406] Lustre: server umount lustre-OST0000 complete [ 1166.818568] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1166.827474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1166.840891] Lustre: Skipped 1 previous similar message [ 1171.941243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1171.947122] Lustre: Skipped 2 previous similar messages [ 1172.480177] Lustre: server umount lustre-MDT0000 complete [ 1177.019919] LustreError: 38556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870617 with bad export cookie 7064689232905263879 [ 1177.024259] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1177.031350] LustreError: 38556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1177.560599] Lustre: server umount lustre-MDT0001 complete [ 1190.096921] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 1193.022450] Key type lgssc unregistered [ 1193.482468] LNet: 42854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1193.490824] LNetError: 42854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1193.523893] LNet: Removed LNI 192.168.202.116@tcp [ 1194.240765] Key type .llcrypt unregistered [ 1194.242500] Key type ._llcrypt unregistered [ 1214.789787] Key type ._llcrypt registered [ 1214.792883] Key type .llcrypt registered [ 1214.904887] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 1234.991779] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1236.049586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1236.106035] alg: No test for adler32 (adler32-zlib) [ 1237.397406] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 1237.695963] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 1239.511380] Key type lgssc registered [ 1240.823315] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1284.156455] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1284.182788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1285.597728] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1285.629054] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1285.740172] Lustre: lustre-MDT0000: new disk, initializing [ 1285.875303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1285.899871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1291.884769] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1303.550262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1303.617038] Lustre: 46012:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1303.658810] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1303.663176] Lustre: Skipped 1 previous similar message [ 1303.814128] Lustre: lustre-MDT0001: new disk, initializing [ 1303.916821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1303.981281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1303.998612] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1308.850163] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1321.409896] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1329.811851] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1338.768267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1339.216958] Lustre: lustre-OST0000: new disk, initializing [ 1339.231067] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1339.246927] Lustre: 47980:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1339.483807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1344.585822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1344.597471] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1344.766254] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1347.286980] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1357.847232] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1362.651875] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1362.896505] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1367.552853] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1367.887855] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1370.609523] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1370.619350] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1370.639446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1370.656319] LustreError: Skipped 1 previous similar message [ 1370.677174] Lustre: Skipped 1 previous similar message [ 1375.717541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1375.728888] Lustre: Skipped 1 previous similar message [ 1376.143347] Lustre: server umount lustre-OST0000 complete [ 1380.832888] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1380.842993] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1380.845163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1380.881084] Lustre: Skipped 2 previous similar messages [ 1385.959983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1385.962633] Lustre: Skipped 1 previous similar message [ 1386.647866] Lustre: server umount lustre-MDT0000 complete [ 1390.341948] LustreError: 46815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870830 with bad export cookie 16967642261418471672 [ 1390.351914] LustreError: 46815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1390.352278] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1390.645425] Lustre: server umount lustre-MDT0001 complete [ 1401.067487] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 08:33:58 (1788870838) [ 1407.508311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1408.155106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1413.600329] LustreError: 50441:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1413.639312] LustreError: 50441:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1414.306242] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1417.702913] LustreError: 50442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1422.816708] LustreError: 50441:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1425.465481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1425.925834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1431.096112] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1439.102959] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1446.048953] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1454.342049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1454.772234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1461.702220] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1470.250357] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1485.123126] LustreError: 52990:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1486.108993] Lustre: Modifying parameter general.timeout=1 in log params [ 1487.150690] Lustre: Modifying parameter general.timeout=2 in log params [ 1490.077479] Lustre: Modifying parameter general.timeout=5 in log params [ 1490.083440] Lustre: Skipped 2 previous similar messages [ 1494.659264] Lustre: Modifying parameter general.timeout=9 in log params [ 1494.661559] Lustre: Skipped 3 previous similar messages [ 1503.217789] Lustre: Modifying parameter general.timeout=18 in log params [ 1503.223111] Lustre: Skipped 8 previous similar messages [ 1519.365467] Lustre: Modifying parameter general.timeout=36 in log params [ 1519.374916] Lustre: Skipped 17 previous similar messages [ 1551.740919] Lustre: Modifying parameter general.timeout=69 in log params [ 1551.744403] Lustre: Skipped 32 previous similar messages [ 1621.540290] Lustre: Disabling parameter general.timeout= in log params [ 1621.542665] Lustre: Skipped 60 previous similar messages [ 1625.992201] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 08:37:42 (1788871062) [ 1629.069981] Lustre: 59946:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1630.912513] Lustre: 60044:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1634.705959] Lustre: 60240:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1634.722672] Lustre: 60240:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1666.239305] Lustre: 61816:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1666.245978] Lustre: 61816:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1666.265234] LustreError: 61816:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1670.756586] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 08:38:28 (1788871108) [ 1686.496608] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1686.515229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1686.519992] Lustre: Skipped 2 previous similar messages [ 1686.531520] Lustre: Skipped 2 previous similar messages [ 1689.667611] Lustre: server umount lustre-MDT0000 complete [ 1691.625244] LustreError: 50442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1691.644569] LustreError: 50442:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1693.628325] LustreError: 50422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788871134 with bad export cookie 16967642261418472554 [ 1693.628895] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1693.637027] LustreError: 50422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1693.914569] Lustre: server umount lustre-MDT0001 complete [ 1708.061669] Lustre: server umount lustre-OST0000 complete [ 1737.130116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1737.415870] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1737.713812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1742.858265] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1751.885287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1751.951289] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1751.978550] Lustre: 64348:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1757.726923] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1765.505075] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1771.492724] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1777.975688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1778.139242] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1778.379777] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1778.387624] Lustre: Skipped 1 previous similar message [ 1783.893721] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1791.858637] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1793.465483] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1793.472644] Lustre: Skipped 23 previous similar messages [ 1793.510535] Lustre: 66914:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1804.768980] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1804.778495] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1804.783861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1804.790622] Lustre: Skipped 1 previous similar message [ 1804.800698] Lustre: Skipped 1 previous similar message [ 1806.537584] Lustre: server umount lustre-OST0000 complete [ 1813.987706] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1813.994675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1825.248583] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1825.259391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1825.280685] Lustre: Skipped 6 previous similar messages [ 1825.492169] Lustre: server umount lustre-MDT0000 complete [ 1829.755298] LustreError: 65089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788871270 with bad export cookie 16967642261418475767 [ 1829.760467] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1829.764068] LustreError: 65089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1830.525026] Lustre: server umount lustre-MDT0001 complete [ 1843.152796] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 1847.011108] Key type lgssc unregistered [ 1847.375848] LNet: 68127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1847.381224] LNetError: 68127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1847.403769] LNet: Removed LNI 192.168.202.116@tcp [ 1848.408582] Key type .llcrypt unregistered [ 1848.414566] Key type ._llcrypt unregistered [ 1877.395335] Key type ._llcrypt registered [ 1877.396634] Key type .llcrypt registered [ 1877.571878] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1879.303390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1879.344147] alg: No test for adler32 (adler32-zlib) [ 1880.490575] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 1880.740586] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 1882.511187] Key type lgssc registered [ 1883.770763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1892.116030] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1892.135931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1893.723087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1899.293915] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1904.097411] LustreError: 69539:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1904.115743] LustreError: 69539:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1908.725641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1909.261671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1914.457794] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1923.393758] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1930.577992] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1938.185463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1938.900643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1946.266797] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1954.817568] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1963.370891] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1974.756848] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1974.782525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1974.791960] Lustre: Skipped 1 previous similar message [ 1979.471662] Lustre: server umount lustre-MDT0000 complete [ 1979.879588] LustreError: 69539:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1979.889603] LustreError: 69539:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1983.860731] LustreError: 69520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788871424 with bad export cookie 5217568369780021158 [ 1983.872721] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1983.876630] LustreError: 69520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1984.233326] Lustre: server umount lustre-MDT0001 complete [ 1999.442706] Lustre: server umount lustre-OST0000 complete [ 2031.841809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2032.264837] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2032.321158] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2032.731173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2038.554837] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2049.917120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2050.096780] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2050.147766] Lustre: 74413:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2050.984543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2057.222409] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2066.385306] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2074.222819] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2082.202092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2082.441438] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2083.153693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2092.357504] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2102.315625] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2104.462761] Lustre: 76990:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2107.206281] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2116.064387] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2116.071632] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2116.080723] Lustre: Skipped 2 previous similar messages [ 2116.087063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2116.583500] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2119.941640] Lustre: server umount lustre-OST0000 complete [ 2126.839317] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2126.862555] Lustre: Skipped 1 previous similar message [ 2126.870622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2126.881021] Lustre: Skipped 1 previous similar message [ 2130.937862] Lustre: server umount lustre-MDT0000 complete [ 2131.939091] LustreError: 74423:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2131.975415] LustreError: 74423:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2135.804891] LustreError: 74404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788871576 with bad export cookie 5217568369780022005 [ 2135.805891] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2135.815376] LustreError: 74404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2136.156782] Lustre: server umount lustre-MDT0001 complete [ 2151.599134] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 2156.406944] Key type lgssc unregistered [ 2157.085542] LNet: 78202:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2157.090964] LNetError: 78202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2157.112400] LNet: Removed LNI 192.168.202.116@tcp [ 2158.834710] Key type .llcrypt unregistered [ 2158.839718] Key type ._llcrypt unregistered [ 2193.426762] Key type ._llcrypt registered [ 2193.429992] Key type .llcrypt registered [ 2193.606821] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2195.010218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2195.059303] alg: No test for adler32 (adler32-zlib) [ 2196.544139] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 2196.828233] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 2198.631181] Key type lgssc registered [ 2199.869257] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2210.202944] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2210.246238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2211.948479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2217.864705] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2222.056208] LustreError: 79614:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2222.072042] LustreError: 79614:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2227.172931] LustreError: 79615:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2228.170347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2228.921549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2235.523256] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2246.235401] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2253.662845] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2263.027651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2263.871855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2271.565180] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2280.431162] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2291.345258] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2292.420379] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2294.522926] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2294.541309] Lustre: Skipped 1 previous similar message [ 2296.789836] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2296.795469] Lustre: Skipped 1 previous similar message [ 2301.269741] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2301.276877] Lustre: Skipped 3 previous similar messages [ 2310.196889] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2310.227641] Lustre: Skipped 7 previous similar messages [ 2327.081650] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2327.084673] Lustre: Skipped 14 previous similar messages [ 2360.213514] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2360.216116] Lustre: Skipped 31 previous similar messages [ 2407.265687] Lustre: 87726:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2410.334400] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 08:50:48 (1788871848) [ 2411.956560] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2414.732076] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 08:50:51 (1788871851) [ 2420.704166] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2420.713782] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2420.718288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2420.738647] Lustre: Skipped 1 previous similar message [ 2420.758225] Lustre: Skipped 1 previous similar message [ 2425.838254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2426.275631] Lustre: server umount lustre-OST0000 complete [ 2430.948851] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2430.971806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2430.984662] Lustre: Skipped 2 previous similar messages [ 2436.066325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2436.074961] Lustre: Skipped 1 previous similar message [ 2437.055390] Lustre: server umount lustre-MDT0000 complete [ 2441.189780] LustreError: 79614:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2441.207031] LustreError: 79614:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2441.298364] LustreError: 82754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788871881 with bad export cookie 8984291168728445211 [ 2441.308737] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2441.311940] LustreError: 82754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2441.694781] Lustre: server umount lustre-MDT0001 complete [ 2455.219513] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 2458.863299] Key type lgssc unregistered [ 2459.241897] LNet: 88975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2459.252529] LNetError: 88975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2459.266431] LNet: Removed LNI 192.168.202.116@tcp [ 2460.289848] Key type .llcrypt unregistered [ 2460.292766] Key type ._llcrypt unregistered [ 2466.047047] Key type ._llcrypt registered [ 2466.051602] Key type .llcrypt registered [ 2466.251925] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2472.324946] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules [ 2473.672587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2473.838275] alg: No test for adler32 (adler32-zlib) [ 2475.296916] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 2475.781493] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 2477.607138] Key type lgssc registered [ 2479.034647] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2480.650355] LustreError: 90223:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2503.656604] LustreError: 90223:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2505.179543] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2518.332712] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2518.368700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2520.122220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2527.058563] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2530.276800] LustreError: 91250:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2530.308548] LustreError: 91250:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2535.392580] LustreError: 91249:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2540.515544] LustreError: 91250:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2540.574110] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 08:52:57 (1788871977) [ 2545.644523] LustreError: 91249:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2546.243556] LustreError: 91281:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2546.260316] LustreError: 91281:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2546.282629] LustreError: 91281:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 26, retries 0, failed: rc = -5 [ 2546.913752] Lustre: server umount lustre-MDT0000 complete [ 2565.089586] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 2568.317197] Key type lgssc unregistered [ 2568.611044] LNet: 92776:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2568.614798] LNetError: 92776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2568.627950] LNet: Removed LNI 192.168.202.116@tcp [ 2569.639766] Key type .llcrypt unregistered [ 2569.642558] Key type ._llcrypt unregistered [ 2592.672417] Key type ._llcrypt registered [ 2592.674432] Key type .llcrypt registered [ 2592.773783] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2593.742983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2593.784423] alg: No test for adler32 (adler32-zlib) [ 2595.099747] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 2595.520496] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 2597.247210] Key type lgssc registered [ 2598.488848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2608.154974] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2608.195133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2609.763076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2615.263349] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2619.895812] LustreError: 93990:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2619.922848] LustreError: 93990:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2624.994771] LustreError: 93989:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2625.216745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2625.705337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2631.769510] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2640.878336] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2647.265446] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2656.922818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2658.026598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2666.813933] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2675.863221] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2730.304962] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 08:56:07 (1788872167) [ 2732.606829] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2734.825476] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 08:56:12 (1788872172) [ 2742.244134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2742.260557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2742.271092] Lustre: Skipped 2 previous similar messages [ 2746.948951] Lustre: server umount lustre-MDT0000 complete [ 2747.363661] LustreError: 96043:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2747.385598] LustreError: 96043:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2751.531644] LustreError: 93971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788872191 with bad export cookie 158106149345256371 [ 2751.537480] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2751.547738] LustreError: 93971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2751.906773] Lustre: server umount lustre-MDT0001 complete [ 2767.466505] Lustre: server umount lustre-OST0000 complete [ 2781.118163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2781.602793] LustreError: 98267:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2781.829848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2787.304797] LustreError: 98268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2787.742084] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2797.544282] LustreError: 98267:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2797.566699] LustreError: 98267:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2798.547381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2799.334687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2804.130399] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2812.880786] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2819.969478] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2835.604173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2835.817838] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2835.822856] LustreError: 98256:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2835.857772] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2835.864875] LustreError: 100352:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2835.872505] LustreError: 100352:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2835.880551] LustreError: 100352:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2835.977960] Lustre: server umount lustre-OST0000 complete [ 2835.982189] LustreError: 100352:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2845.426796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2845.654390] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2845.669249] LustreError: 98257:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2845.697840] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2845.714837] LustreError: 100819:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2845.726329] LustreError: 100819:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2845.732696] LustreError: 100819:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2845.815749] Lustre: server umount lustre-OST0000 complete [ 2845.818627] LustreError: 100819:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2857.273058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2857.496512] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2857.501700] Lustre: Found index 0 for lustre-OST0000, updating log [ 2857.511685] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2857.658559] Lustre: lustre-OST0000: new disk, initializing [ 2857.674895] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2857.687402] Lustre: 101354:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2858.161511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2866.460151] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2871.129298] Lustre: Failing over lustre-OST0000 [ 2871.427872] Lustre: server umount lustre-OST0000 complete [ 2879.971243] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2879.979090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2879.987664] Lustre: Skipped 3 previous similar messages [ 2880.011965] Lustre: Skipped 1 previous similar message [ 2882.589486] Lustre: server umount lustre-MDT0000 complete [ 2885.089398] LustreError: 98268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2885.110415] LustreError: 98268:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2887.397462] LustreError: 98247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788872327 with bad export cookie 158106149345259115 [ 2887.404550] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2887.411501] LustreError: 98247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2897.166877] Lustre: server umount lustre-MDT0001 complete [ 2907.893686] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 08:59:05 (1788872345) [ 2928.735467] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2941.190815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2941.672903] LustreError: 103723:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2941.771020] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2946.478671] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2955.698141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2961.307754] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2964.514218] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2971.065506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2971.867628] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2978.033530] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2981.099721] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2981.139636] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2981.147029] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2987.653280] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2987.727381] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2987.856153] Lustre: lustre-OST0001: new disk, initializing [ 2987.858741] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2987.862298] Lustre: 106039:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2988.179696] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2988.187216] Lustre: Skipped 2 previous similar messages [ 2995.757503] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2996.892437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2996.903419] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2997.033540] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3004.320950] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3010.483807] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3017.701441] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3017.716936] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3017.720561] Lustre: Skipped 2 previous similar messages [ 3017.741136] Lustre: Skipped 2 previous similar messages [ 3020.783816] Lustre: server umount lustre-MDT0001 complete [ 3022.815832] LustreError: 103723:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3022.830149] LustreError: 103723:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3031.589122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3031.746840] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3031.750885] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3031.800690] Lustre: 103711:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3033.786881] LustreError: 107624:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3033.802905] LustreError: 107624:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3033.825918] LustreError: 103715:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3033.975956] LustreError: 107636:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.116@tcp: cfg command failed: rc = -17 [ 3033.983868] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3033.983868] [ 3037.121360] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3037.684283] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3047.732174] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 09:01:25 (1788872485) [ 3066.163358] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3083.966643] Lustre: 109665:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3106.931806] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3111.212709] Lustre: 110800:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3195.872291] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3195.882473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3195.907875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3195.919666] Lustre: Skipped 1 previous similar message [ 3201.504779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3201.516265] Lustre: Skipped 7 previous similar messages [ 3201.712372] Lustre: server umount lustre-MDT0000 complete [ 3205.469370] LustreError: 104862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788872645 with bad export cookie 158106149345260116 [ 3205.471706] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3205.474304] LustreError: 104862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3205.790220] Lustre: server umount lustre-MDT0001 complete [ 3220.409222] Lustre: server umount lustre-OST0000 complete [ 3222.751110] Lustre: 93302:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788872647/real 1788872647] req@ffff96b888354380 x1875768283334784/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788872663 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3222.781333] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3222.800062] Lustre: Skipped 3 previous similar messages [ 3225.324447] Lustre: server umount lustre-OST0001 complete [ 3231.725156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3242.721864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3252.322054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3262.642798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3281.720064] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3292.436955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3292.466217] Lustre: lustre-MDT0000: reset Object Index mappings [ 3293.004475] LustreError: 114756:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3293.025592] LustreError: 114756:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 3293.149575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3293.158588] Lustre: Skipped 1 previous similar message [ 3298.077381] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3307.683256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3307.718255] Lustre: lustre-MDT0001: reset Object Index mappings [ 3313.985935] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3317.451404] Lustre: 115903:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3325.671562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3332.915357] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:226 to 0x280000400:257) [ 3334.669387] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3338.065910] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3345.779814] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3352.092748] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3352.112231] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3354.048540] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3363.694451] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3368.222508] Lustre: 117781:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3425.665840] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 09:07:43 (1788872863) [ 3451.891065] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3451.895258] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3451.915221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3457.901857] Lustre: server umount lustre-MDT0000 complete [ 3461.753929] LustreError: 114737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788872902 with bad export cookie 158106149345306981 [ 3461.758505] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3461.768265] LustreError: 114737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3468.231151] Lustre: server umount lustre-MDT0001 complete [ 3479.059342] Lustre: server umount lustre-OST0000 complete [ 3490.582922] Lustre: server umount lustre-OST0001 complete [ 3498.034642] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 3511.272756] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3556.640356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3556.772432] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3556.789947] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3556.856485] Lustre: lustre-MDT0000: new disk, initializing [ 3556.908432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3556.911755] Lustre: Skipped 3 previous similar messages [ 3556.924733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3562.029512] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3570.584903] Lustre: server umount lustre-MDT0000 complete [ 3582.403533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3582.806364] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3582.835329] Lustre: 122056:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3582.857950] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3588.406966] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3594.988353] Lustre: server umount lustre-MDT0000 complete [ 3606.398873] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 09:10:43 (1788873043) [ 3608.449128] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3611.760545] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 09:10:48 (1788873048) [ 3631.207561] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3642.324234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3646.993240] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3656.484638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3656.695293] Lustre: 124107:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3656.839209] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3656.844809] Lustre: Skipped 1 previous similar message [ 3656.951573] Lustre: lustre-MDT0001: new disk, initializing [ 3657.119906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3657.144771] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3662.643222] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3668.716073] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3676.287937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3676.566854] Lustre: lustre-OST0000: new disk, initializing [ 3676.570252] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3676.575299] Lustre: 125686:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3678.147372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3678.160511] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3678.325050] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3683.327296] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3694.739506] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3694.988328] Lustre: lustre-OST0001: new disk, initializing [ 3694.991376] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3694.996811] Lustre: 126556:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3696.600851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3696.613211] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3696.711331] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3701.996247] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3713.513909] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3718.939650] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4158.267448] Lustre: 132978:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4158.273279] Lustre: 132978:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4158.276856] Lustre: 132978:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4158.283399] Lustre: 132978:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4158.288701] Lustre: 132978:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4158.296627] Lustre: 132978:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4444.948680] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:24:42 (1788873882) [ 4459.499422] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4459.513313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4459.515630] Lustre: Skipped 8 previous similar messages [ 4459.531367] Lustre: Skipped 15 previous similar messages [ 4464.619219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4464.622664] Lustre: Skipped 3 previous similar messages [ 4469.730485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4473.311173] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4473.521612] Lustre: server umount lustre-MDT0000 complete [ 4474.863391] LustreError: 124118:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4474.890834] LustreError: 124118:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 4476.799725] LustreError: 124099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788873917 with bad export cookie 158106149345376316 [ 4476.805629] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4476.818180] LustreError: 124099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4479.977078] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4479.979719] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4479.996718] Lustre: Skipped 1 previous similar message [ 4480.002391] Lustre: Skipped 4 previous similar messages [ 4491.232279] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4491.672330] Lustre: server umount lustre-MDT0001 complete [ 4501.442403] Lustre: server umount lustre-OST0000 complete [ 4519.391146] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4519.660682] Lustre: server umount lustre-OST0001 complete [ 4525.603675] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 4534.794242] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 4570.030741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4570.273760] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4570.295892] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4570.357785] Lustre: lustre-MDT0000: new disk, initializing [ 4570.422196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4570.425291] Lustre: Skipped 5 previous similar messages [ 4570.435255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4575.044092] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4584.739649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4584.869486] Lustre: 136036:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4584.937091] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4584.945127] Lustre: Skipped 1 previous similar message [ 4585.088866] Lustre: lustre-MDT0001: new disk, initializing [ 4585.189940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4585.204414] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4588.966864] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4598.452177] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4604.045194] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4610.909879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4611.138409] Lustre: lustre-OST0000: new disk, initializing [ 4611.142845] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4611.147136] Lustre: 138003:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4611.223983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4611.226866] Lustre: Skipped 1 previous similar message [ 4612.657791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4612.668710] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4612.833157] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4617.311217] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4625.775959] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4634.485569] Lustre: lustre-MDD0000: changelog on [ 4636.914446] Lustre: lustre-MDD0001: changelog on [ 6429.663326] Lustre: lustre-MDD0001: changelog off [ 6431.098723] Lustre: 138818:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6431.854738] Lustre: lustre-MDD0000: changelog off [ 6440.938275] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6440.948462] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6440.953350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6440.953363] Lustre: Skipped 4 previous similar messages [ 6440.957393] LustreError: Skipped 1 previous similar message [ 6440.969109] Lustre: Skipped 1 previous similar message [ 6442.566302] Lustre: server umount lustre-OST0000 complete [ 6446.050090] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6446.054389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6446.054566] Lustre: Skipped 1 previous similar message [ 6446.064296] Lustre: Skipped 2 previous similar messages [ 6451.171193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6451.178409] Lustre: Skipped 1 previous similar message [ 6451.877828] Lustre: server umount lustre-MDT0000 complete [ 6455.342237] LustreError: 136027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788875895 with bad export cookie 158106149346197535 [ 6455.347358] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6455.353222] LustreError: 136027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6455.589035] Lustre: server umount lustre-MDT0001 complete [ 6466.969457] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 6469.515226] Key type lgssc unregistered [ 6469.749786] LNet: 147138:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6469.760280] LNetError: 147138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6469.776465] LNet: Removed LNI 192.168.202.116@tcp [ 6470.457198] Key type .llcrypt unregistered [ 6470.459066] Key type ._llcrypt unregistered [ 6484.735303] Key type ._llcrypt registered [ 6484.737628] Key type .llcrypt registered [ 6484.806194] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 6495.358257] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 6496.087242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6496.119025] alg: No test for adler32 (adler32-zlib) [ 6497.130613] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 6497.347181] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 6498.976495] Key type lgssc registered [ 6499.887647] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6521.241817] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6521.250977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6522.456821] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6522.489381] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6522.541060] Lustre: lustre-MDT0000: new disk, initializing [ 6522.587567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6522.601894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6525.529519] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6532.581604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6532.634500] Lustre: 150311:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6532.670843] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6532.674327] Lustre: Skipped 1 previous similar message [ 6532.727403] Lustre: lustre-MDT0001: new disk, initializing [ 6532.783731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6532.810422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6532.816678] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6535.170404] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6541.409930] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6545.380608] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6550.325562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6550.485661] Lustre: lustre-OST0000: new disk, initializing [ 6550.488469] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6550.492242] Lustre: 152275:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6550.562877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6554.613384] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6558.312103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6558.321616] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6558.468546] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6562.056716] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6567.190487] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_module obdecho/obdecho [ 6568.682471] LustreError: 153280:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6568.688210] LustreError: 153280:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6570.060618] LustreError: 153376:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6579.170873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6579.174515] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6579.179337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6579.196820] Lustre: Skipped 2 previous similar messages [ 6582.686988] Lustre: server umount lustre-MDT0000 complete [ 6585.295690] LustreError: 150303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876025 with bad export cookie 16192613418082687489 [ 6585.300561] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6585.304214] LustreError: 150303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6585.586399] Lustre: server umount lustre-MDT0001 complete [ 6597.800587] Lustre: server umount lustre-OST0000 complete [ 6612.500777] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 6618.774691] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 6644.508191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6644.703325] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6644.751665] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6644.835120] Lustre: lustre-MDT0000: new disk, initializing [ 6644.878684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6644.891479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6648.234678] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6656.232243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6656.338548] Lustre: 157102:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6656.391688] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6656.394585] Lustre: Skipped 1 previous similar message [ 6656.464469] Lustre: lustre-MDT0001: new disk, initializing [ 6656.535331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6656.568721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6656.582263] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6660.114755] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6667.339255] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6670.871819] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6676.131775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6676.421263] Lustre: lustre-OST0000: new disk, initializing [ 6676.423576] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6676.429388] Lustre: 159071:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6676.495863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6680.918991] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6682.156286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6682.164128] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6682.268847] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6687.913554] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6691.591198] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6691.782962] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6695.762128] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6695.972705] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6697.954917] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6697.962385] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6697.969358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6697.971758] Lustre: Skipped 1 previous similar message [ 6697.982726] Lustre: Skipped 3 previous similar messages [ 6703.082475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6703.086952] Lustre: Skipped 1 previous similar message [ 6703.677254] Lustre: server umount lustre-OST0000 complete [ 6707.680143] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6707.684092] LustreError: Skipped 1 previous similar message [ 6707.687486] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6707.704645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6712.837587] Lustre: server umount lustre-MDT0000 complete [ 6713.319888] LustreError: 157114:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6713.332326] LustreError: 157114:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6715.289390] LustreError: 159070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876155 with bad export cookie 16192613418082688448 [ 6715.300692] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6715.492241] Lustre: server umount lustre-MDT0001 complete [ 6718.372904] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 10:02:36 (1788876156) [ 6719.564787] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6720.926674] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 10:02:39 (1788876159) [ 6726.913574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6727.336748] LustreError: 161480:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6727.392555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6730.612788] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6732.771145] LustreError: 161481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6736.870817] LustreError: 161480:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6737.863989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6741.614107] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6746.931460] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6751.048463] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6755.768419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6760.089387] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6765.063879] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6780.976802] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.16@tcp (stopping) [ 6780.981929] Lustre: Skipped 2 previous similar messages [ 6782.527164] Lustre: server umount lustre-MDT0000 complete [ 6782.949497] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6782.954298] LustreError: 161477:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6782.960460] Lustre: Skipped 3 previous similar messages [ 6782.966402] LustreError: 161477:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6785.336566] LustreError: 161463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876225 with bad export cookie 16192613418082689330 [ 6785.347978] LustreError: 161463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6785.353347] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6788.066212] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6790.112684] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6790.115143] Lustre: Skipped 2 previous similar messages [ 6791.197535] LustreError: 161476:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6791.216067] LustreError: 161476:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6791.706876] Lustre: server umount lustre-MDT0001 complete [ 6796.962707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6806.056689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6806.258742] LustreError: 165337:0:(llog.c:1655:llog_backup()) MGC192.168.202.116@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6806.268638] Lustre: 165337:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.116@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6814.688803] LustreError: 165342:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6847.059872] LustreError: 165344:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6847.073205] LustreError: 165344:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 28 previous similar messages [ 6867.423268] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6867.428633] Lustre: 165337:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.116@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6907.959321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6907.963461] Lustre: Skipped 2 previous similar messages [ 6910.001027] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6911.830575] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6912.029844] LustreError: 165344:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6912.043704] LustreError: 165344:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 43 previous similar messages [ 6919.701558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6930.408192] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe0b7b81c920afa6f [ 6930.424757] Lustre: MGC192.168.202.116@tcp: Connection restored to 0@lo (at 0@lo) [ 6934.096488] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6936.046551] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6940.321591] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6941.561943] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6946.289410] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6947.674446] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6961.633238] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6961.637201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6961.641066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6961.644144] Lustre: Skipped 2 previous similar messages [ 6961.657210] Lustre: Skipped 3 previous similar messages [ 6967.593132] Lustre: server umount lustre-MDT0000 complete [ 6970.369886] LustreError: 161462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876410 with bad export cookie 16192613418082695791 [ 6970.371410] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6970.378356] LustreError: 161462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6970.615600] Lustre: server umount lustre-MDT0001 complete [ 6982.909810] Lustre: server umount lustre-OST0000 complete [ 7000.031597] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 7008.479542] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 7036.050650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7036.230653] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7036.251994] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7036.320516] Lustre: lustre-MDT0000: new disk, initializing [ 7036.387456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7036.391777] Lustre: Skipped 1 previous similar message [ 7036.407316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7040.062913] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7050.100106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7050.242889] Lustre: 170441:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7050.545646] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7054.222989] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7061.515773] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7065.560336] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7070.130329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7070.294992] Lustre: lustre-OST0000: new disk, initializing [ 7070.297167] Lustre: Skipped 1 previous similar message [ 7070.299690] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7070.302869] Lustre: Skipped 2 previous similar messages [ 7070.306345] Lustre: 172412:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7074.232654] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7076.428144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7076.436484] Lustre: Skipped 1 previous similar message [ 7076.446512] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7076.543902] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7080.976795] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7084.633468] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7084.801618] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7087.741319] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7087.879866] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7092.197350] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7092.197377] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7092.211447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7092.215048] Lustre: Skipped 4 previous similar messages [ 7095.312692] Lustre: server umount lustre-OST0000 complete [ 7101.920952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7101.927825] LustreError: Skipped 1 previous similar message [ 7112.159183] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7112.277516] Lustre: server umount lustre-MDT0000 complete [ 7114.863892] LustreError: 170433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876555 with bad export cookie 16192613418082712619 [ 7114.868151] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7114.873351] LustreError: 170433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7115.080690] Lustre: server umount lustre-MDT0001 complete [ 7121.716857] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 10:09:20 (1788876560) [ 7126.792188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7127.067856] LustreError: 174822:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7127.077272] LustreError: 174822:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 7130.545675] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7136.783662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7139.822308] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7145.355729] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7149.592552] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7154.480752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7159.397351] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7164.676258] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7182.807380] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 10:10:21 (1788876621) [ 7187.430675] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7187.435158] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7187.442622] Lustre: Skipped 3 previous similar messages [ 7187.446625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7187.451757] Lustre: Skipped 7 previous similar messages [ 7191.803420] Lustre: server umount lustre-OST0000 complete [ 7200.786264] Lustre: server umount lustre-MDT0000 complete [ 7203.209427] LustreError: 174804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876643 with bad export cookie 16192613418082713501 [ 7203.217588] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7203.407276] Lustre: server umount lustre-MDT0001 complete [ 7210.637953] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 7212.481546] Key type lgssc unregistered [ 7212.685622] LNet: 178567:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7212.692181] LNetError: 178567:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7212.705169] LNet: Removed LNI 192.168.202.116@tcp [ 7213.228615] Key type .llcrypt unregistered [ 7213.230453] Key type ._llcrypt unregistered [ 7226.769244] Key type ._llcrypt registered [ 7226.771188] Key type .llcrypt registered [ 7226.862377] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 7227.751017] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7227.764225] alg: No test for adler32 (adler32-zlib) [ 7228.766607] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 7228.984762] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 7230.623164] Key type lgssc registered [ 7231.379440] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7236.553541] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7236.569815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7252.065654] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7257.183331] LustreError: 179800:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.116@tcp: failed processing log, type 4: rc = -110 [ 7262.305317] LustreError: 179800:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7262.312355] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7262.318948] LustreError: MGC192.168.202.116@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7262.330397] LustreError: 179800:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7262.340165] LustreError: 179800:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -22 [ 7262.346626] LustreError: 179800:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 7262.391305] Lustre: server umount lustre-OST0000 complete [ 7262.392883] LustreError: 179800:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7268.700335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7270.014728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7273.221692] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7279.717792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7279.950175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7283.185216] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7288.965179] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7293.409466] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7298.835234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7299.026848] LustreError: 182118:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7299.035330] Lustre: 182118:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.116@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7299.154322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7303.978941] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7309.691848] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7311.375054] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7311.379591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7311.382348] Lustre: Skipped 2 previous similar messages [ 7311.393388] Lustre: Skipped 2 previous similar messages [ 7316.448360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7316.452130] Lustre: Skipped 2 previous similar messages [ 7317.673170] Lustre: server umount lustre-MDT0000 complete [ 7320.239652] LustreError: 181036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876760 with bad export cookie 12081031929590006380 [ 7320.241311] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7320.245241] LustreError: 181036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7320.473420] Lustre: server umount lustre-MDT0001 complete [ 7333.317112] Lustre: server umount lustre-OST0000 complete [ 7339.749284] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 7347.440931] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 7373.327768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7373.498593] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7373.521382] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7373.564447] Lustre: lustre-MDT0000: new disk, initializing [ 7373.663870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7373.677111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7377.313965] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7386.022614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7386.105099] Lustre: 185640:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7386.139210] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7386.142455] Lustre: Skipped 1 previous similar message [ 7386.190545] Lustre: lustre-MDT0001: new disk, initializing [ 7386.274037] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7386.310416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7386.325299] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7389.901711] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7397.699989] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7402.297752] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7407.434176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7407.624254] Lustre: lustre-OST0000: new disk, initializing [ 7407.627034] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7407.633694] Lustre: 187607:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7407.680498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7413.297079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7413.303748] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7413.381057] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7413.481576] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7421.677612] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7425.414714] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7425.563669] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7429.350531] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7429.567912] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7433.701530] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7433.702590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7433.717319] Lustre: Skipped 1 previous similar message [ 7433.725860] Lustre: Skipped 1 previous similar message [ 7437.481889] Lustre: server umount lustre-OST0000 complete [ 7442.917129] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7442.924220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7442.939188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7446.635116] Lustre: server umount lustre-MDT0000 complete [ 7449.056247] LustreError: 185645:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7449.065379] LustreError: 185645:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7449.672341] LustreError: 185632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876890 with bad export cookie 12081031929590007101 [ 7449.673594] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7449.680962] LustreError: 185632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7449.917654] Lustre: server umount lustre-MDT0001 complete [ 7456.153059] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 10:14:54 (1788876894) [ 7468.000964] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 7469.856294] Key type lgssc unregistered [ 7470.034495] LNet: 190546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7470.043830] LNetError: 190546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7470.057706] LNet: Removed LNI 192.168.202.116@tcp [ 7470.528421] Key type .llcrypt unregistered [ 7470.529679] Key type ._llcrypt unregistered [ 7486.710830] Key type ._llcrypt registered [ 7486.712961] Key type .llcrypt registered [ 7486.828400] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 7487.719949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7487.818345] alg: No test for adler32 (adler32-zlib) [ 7488.848654] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 7489.086107] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 7490.735427] Key type lgssc registered [ 7491.928416] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7498.741873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7498.754597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7500.121865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7503.682722] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7510.507267] LustreError: 191763:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7510.523404] LustreError: 191763:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7510.682172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7510.911384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7514.481830] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7520.462355] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7524.463259] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7529.804601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7529.961247] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7529.965642] LustreError: 193626:0:(llog.c:1696:llog_backup()) MGC192.168.202.116@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7529.974954] Lustre: 193626:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.116@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7530.106947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7534.444644] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7539.245068] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7541.730055] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7541.737077] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7541.745064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7541.752602] Lustre: Skipped 2 previous similar messages [ 7541.760119] Lustre: Skipped 1 previous similar message [ 7546.852671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7551.971345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7551.979035] Lustre: Skipped 4 previous similar messages [ 7555.039236] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7555.181575] Lustre: server umount lustre-MDT0000 complete [ 7557.091306] LustreError: 191782:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7557.108017] LustreError: 191782:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7557.950295] LustreError: 192498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876998 with bad export cookie 1201885155725500004 [ 7557.960405] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7558.211314] Lustre: server umount lustre-MDT0001 complete [ 7571.531206] Lustre: server umount lustre-OST0000 complete [ 7578.781309] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 7586.529362] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 7612.112434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7612.285558] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7612.306443] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7612.354359] Lustre: lustre-MDT0000: new disk, initializing [ 7612.421855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7612.434698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7615.662710] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7623.307345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7623.422203] Lustre: 197151:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7623.454629] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7623.457156] Lustre: Skipped 1 previous similar message [ 7623.458800] LustreError: 197933:0:(llog.c:1696:llog_backup()) MGC192.168.202.116@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7623.462372] LustreError: 197933:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7623.465082] Lustre: 197933:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.116@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7623.479573] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7623.482738] Lustre: Skipped 1 previous similar message [ 7623.531445] Lustre: lustre-MDT0001: new disk, initializing [ 7623.594846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7623.631393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7623.643970] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7626.709614] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7633.774963] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7637.675499] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7642.123514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7642.321313] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7642.323458] Lustre: Skipped 1 previous similar message [ 7642.325560] LustreError: 199095:0:(llog.c:1696:llog_backup()) MGC192.168.202.116@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7642.334846] LustreError: 199095:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7642.338172] Lustre: 199095:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.116@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7642.342464] Lustre: 199095:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7642.358884] Lustre: lustre-OST0000: new disk, initializing [ 7642.361367] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7642.365217] Lustre: 199120:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7642.440184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7646.951375] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7651.354793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7651.363469] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7651.474692] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7653.669802] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7656.611841] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7656.765507] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7659.562989] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7659.723491] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7661.549901] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7661.555812] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7661.566467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7661.570341] Lustre: Skipped 1 previous similar message [ 7666.660940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7666.670586] Lustre: Skipped 1 previous similar message [ 7675.359465] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7675.452157] Lustre: server umount lustre-OST0000 complete [ 7679.974073] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7679.977993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7679.985020] Lustre: Skipped 1 previous similar message [ 7679.991299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7679.995839] Lustre: Skipped 2 previous similar messages [ 7682.018980] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7692.255253] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7692.396763] Lustre: server umount lustre-MDT0000 complete [ 7695.227396] LustreError: 197144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877135 with bad export cookie 1201885155725500718 [ 7695.236964] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7695.239370] LustreError: 197144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7695.442706] Lustre: server umount lustre-MDT0001 complete [ 7701.313567] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 10:18:59 (1788877139) [ 7711.357453] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 7717.705311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7717.960888] LustreError: 202242:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7718.009423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7720.894737] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7723.494504] LustreError: 202243:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7726.465891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7729.480298] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7731.391218] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7735.375949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7739.391393] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7744.968659] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7745.104447] Lustre: lustre-OST0001: new disk, initializing [ 7745.108939] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7745.115316] Lustre: 204545:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7747.244334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7747.254997] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7747.415381] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7749.520203] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7758.085868] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7762.685965] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7764.119332] Lustre: DEBUG MARKER: ADD OST3 [ 7767.589939] loop: module loaded [ 7767.647162] loop0: detected capacity change from 0 to 204800000 [ 7767.781275] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7767.791325] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7767.806600] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7772.211948] loop0: detected capacity change from 0 to 204800000 [ 7773.928230] Lustre: DEBUG MARKER: STOP OST3 [ 7776.062543] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7776.169210] Lustre: lustre-OST0003: new disk, initializing [ 7776.171114] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7776.173845] Lustre: 206474:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7776.247508] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7776.250951] Lustre: Skipped 3 previous similar messages [ 7777.681323] Lustre: Failing over lustre-OST0003 [ 7777.810857] Lustre: server umount lustre-OST0003 complete [ 7777.977505] LustreError: 203727:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7780.303157] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7785.763540] Lustre: DEBUG MARKER: START OST3 again [ 7788.053840] LustreError: 203728:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7788.065489] LustreError: 203728:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 7789.928441] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7793.490786] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7795.204988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7795.213794] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7795.246827] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 7810.682414] Lustre: Failing over lustre-OST0003 [ 7810.740426] Lustre: server umount lustre-OST0003 complete [ 7815.659871] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7815.672429] Lustre: Skipped 1 previous similar message [ 7815.674516] LustreError: 203727:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7815.677621] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7815.696363] LustreError: 203727:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7825.890505] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7825.892848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7825.904818] Lustre: Skipped 2 previous similar messages [ 7825.919356] Lustre: Skipped 9 previous similar messages [ 7831.541794] Lustre: server umount lustre-MDT0000 complete [ 7833.927342] LustreError: 205371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877274 with bad export cookie 1201885155725501600 [ 7833.933897] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7833.942760] LustreError: 205371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7840.344721] Lustre: server umount lustre-MDT0001 complete [ 7843.232325] Lustre: server umount lustre-OST0000 complete [ 7845.550918] Lustre: server umount lustre-OST0001 complete [ 7849.953069] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 7855.405711] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 7878.564662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7878.734427] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7878.755989] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7878.812820] Lustre: lustre-MDT0000: new disk, initializing [ 7878.871039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7878.877684] Lustre: Skipped 1 previous similar message [ 7878.887275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7881.381736] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7888.338679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7888.405804] Lustre: 211846:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7888.558577] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7890.796713] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7896.843335] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7900.223064] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7904.338637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7904.497413] Lustre: lustre-OST0000: new disk, initializing [ 7904.499831] Lustre: Skipped 1 previous similar message [ 7904.502641] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7904.505986] Lustre: Skipped 2 previous similar messages [ 7904.512597] Lustre: 213812:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7906.212758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7906.220534] Lustre: Skipped 1 previous similar message [ 7906.223485] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7906.287830] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7907.972746] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7914.080220] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7916.313923] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7916.463988] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7919.161525] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7919.250085] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7921.632849] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7921.633645] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7921.639947] LustreError: Skipped 1 previous similar message [ 7921.640589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7921.648036] Lustre: Skipped 4 previous similar messages [ 7921.649747] Lustre: Skipped 8 previous similar messages [ 7926.670950] Lustre: server umount lustre-OST0000 complete [ 7929.826437] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7935.107625] Lustre: server umount lustre-MDT0000 complete [ 7936.995751] LustreError: 211858:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7937.010418] LustreError: 211858:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 7937.221398] LustreError: 213822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877377 with bad export cookie 1201885155725503875 [ 7937.227377] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7937.228864] LustreError: 213822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7937.435393] Lustre: server umount lustre-MDT0001 complete [ 7939.451081] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 10:22:58 (1788877378) [ 7949.157244] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 7954.775240] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 7974.578809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7974.799337] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7974.836407] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7974.891410] Lustre: lustre-MDT0000: new disk, initializing [ 7974.953871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7977.479898] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7984.207181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7984.264688] Lustre: 218417:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7984.431119] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7987.091776] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7993.134951] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7996.517418] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8000.766348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8000.933690] Lustre: 220382:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8002.170840] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8005.333362] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8011.814377] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8015.004111] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8015.165534] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8017.823283] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8018.014569] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8022.503066] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8022.504648] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8022.505887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8022.505894] Lustre: Skipped 3 previous similar messages [ 8022.512931] Lustre: Skipped 3 previous similar messages [ 8025.487677] Lustre: server umount lustre-OST0000 complete [ 8033.701949] Lustre: server umount lustre-MDT0000 complete [ 8035.493326] LustreError: 218410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877475 with bad export cookie 1201885155725504757 [ 8035.498258] LustreError: 218410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8035.499175] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8035.636869] Lustre: server umount lustre-MDT0001 complete [ 8039.609218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8039.846264] LustreError: 222598:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8039.878138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8039.881315] Lustre: Skipped 5 previous similar messages [ 8041.863321] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8045.818877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8047.852222] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8051.676312] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8055.023155] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8059.160338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8062.618555] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8066.606088] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8075.758248] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8078.012941] Lustre: server umount lustre-OST0000 complete [ 8098.274813] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8098.690275] Lustre: server umount lustre-MDT0000 complete [ 8104.426734] LustreError: 222579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877544 with bad export cookie 1201885155725505639 [ 8104.427650] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8104.436198] LustreError: 222579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8105.928905] Lustre: server umount lustre-MDT0001 complete [ 8207.940532] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 8211.327962] Key type lgssc unregistered [ 8211.827793] LNet: 226266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8211.833386] LNetError: 226266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8211.846246] LNet: Removed LNI 192.168.202.116@tcp [ 8212.684347] Key type .llcrypt unregistered [ 8212.686885] Key type ._llcrypt unregistered [ 8239.366890] Key type ._llcrypt registered [ 8239.373900] Key type .llcrypt registered [ 8239.506160] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 8255.427475] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 8256.882969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8256.995702] alg: No test for adler32 (adler32-zlib) [ 8258.143940] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 8258.391736] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 8260.079258] Key type lgssc registered [ 8261.642857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8306.989698] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8307.030603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8308.501555] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8308.550051] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8308.764774] Lustre: lustre-MDT0000: new disk, initializing [ 8308.950669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8308.988895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8315.565276] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8326.943412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8327.101510] Lustre: 229616:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8327.160739] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8327.164941] Lustre: Skipped 1 previous similar message [ 8327.256671] Lustre: lustre-MDT0001: new disk, initializing [ 8327.377290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8327.412647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8327.430291] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8332.368594] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8342.069604] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8349.587968] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8357.314430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8357.596593] Lustre: lustre-OST0000: new disk, initializing [ 8357.599882] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8357.604548] Lustre: 231592:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8357.693856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8363.107628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8363.120640] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8363.273103] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8365.272383] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8376.745956] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8383.259981] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8383.533191] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8388.794926] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8389.103848] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8394.209797] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8394.227288] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8394.247101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8394.263843] Lustre: Skipped 1 previous similar message [ 8397.665208] Lustre: server umount lustre-OST0000 complete [ 8404.450082] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8404.452450] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8404.453894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8404.478039] Lustre: Skipped 2 previous similar messages [ 8404.501069] LustreError: Skipped 1 previous similar message [ 8408.640333] Lustre: server umount lustre-MDT0000 complete [ 8412.685720] LustreError: 229610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877853 with bad export cookie 2231641756789283771 [ 8412.701032] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8412.710817] LustreError: 229610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8413.143926] Lustre: server umount lustre-MDT0001 complete [ 8435.820263] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 8450.256502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8450.601555] LustreError: 234528:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8450.609733] LustreError: 234528:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8450.632896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8455.652826] LustreError: 234529:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8456.041242] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8459.755527] LustreError: 234528:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8464.729168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8464.871026] LustreError: 234529:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8465.098273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8469.503833] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8472.189660] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8478.162399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8478.407772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8484.388255] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8491.706506] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8491.849942] Lustre: lustre-OST0001: new disk, initializing [ 8491.852866] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8491.858195] Lustre: 236837:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8498.420479] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8500.356248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8500.372307] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8500.465536] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8511.082940] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8518.407625] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8532.277542] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 10:32:49 (1788877969) [ 8536.553819] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8536.564918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8536.571682] Lustre: Skipped 2 previous similar messages [ 8536.578588] Lustre: Skipped 1 previous similar message [ 8540.707808] Lustre: server umount lustre-MDT0000 complete [ 8541.674069] LustreError: 234525:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8541.693328] LustreError: 234525:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8544.505344] LustreError: 234509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877984 with bad export cookie 2231641756789284653 [ 8544.508042] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8544.518530] LustreError: 234509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8544.760750] Lustre: server umount lustre-MDT0001 complete [ 8557.972375] Lustre: server umount lustre-OST0000 complete [ 8572.362917] Lustre: server umount lustre-OST0001 complete [ 8579.472738] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 8589.709878] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 8625.593288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8625.804500] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8625.823267] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8625.874993] Lustre: lustre-MDT0000: new disk, initializing [ 8625.938843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8625.943471] Lustre: Skipped 1 previous similar message [ 8625.954929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8630.632257] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8642.193612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8642.262666] Lustre: 240785:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8642.288193] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8642.290184] Lustre: Skipped 1 previous similar message [ 8642.335771] Lustre: lustre-MDT0001: new disk, initializing [ 8642.435178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8642.451660] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8647.320391] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8656.681568] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8664.242955] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8672.163897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8672.459227] Lustre: lustre-OST0000: new disk, initializing [ 8672.462268] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8672.466659] Lustre: 242752:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8672.584186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8672.589229] Lustre: Skipped 1 previous similar message [ 8674.558317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8674.588578] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8674.811533] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8678.104479] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8687.135390] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8691.990315] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8692.164448] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8695.864800] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8696.025068] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8700.385650] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8700.389204] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8700.392477] LustreError: Skipped 1 previous similar message [ 8700.400606] Lustre: Skipped 1 previous similar message [ 8700.403068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8700.403074] Lustre: Skipped 3 previous similar messages [ 8703.766806] Lustre: server umount lustre-OST0000 complete [ 8709.088467] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8709.093061] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8709.103542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8709.110944] Lustre: Skipped 1 previous similar message [ 8712.940460] Lustre: server umount lustre-MDT0000 complete [ 8715.473147] LustreError: 241589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788878155 with bad export cookie 2231641756789286788 [ 8715.476967] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8715.477621] LustreError: 241589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8715.658503] Lustre: server umount lustre-MDT0001 complete [ 8721.434705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8721.725843] LustreError: 244967:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8725.060123] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8731.028165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8733.591586] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8738.351114] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8742.371787] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8746.898183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8747.101617] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8747.105703] Lustre: Skipped 2 previous similar messages [ 8751.825489] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8756.318315] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8902.111923] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8902.114502] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8902.118607] Lustre: Skipped 1 previous similar message [ 8902.120689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8902.123545] Lustre: Skipped 2 previous similar messages [ 8906.320294] Lustre: server umount lustre-OST0000 complete [ 8910.306697] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8910.312382] LustreError: Skipped 1 previous similar message [ 8920.544753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8920.548691] Lustre: Skipped 7 previous similar messages [ 8922.079139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8922.141698] Lustre: server umount lustre-MDT0000 complete [ 8923.688260] LustreError: 244949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788878364 with bad export cookie 2231641756789287663 [ 8923.691962] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8923.692159] LustreError: 244949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8923.821986] Lustre: server umount lustre-MDT0001 complete [ 9021.654152] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 9022.745304] Key type lgssc unregistered [ 9022.874360] LNet: 248573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9022.877130] LNetError: 248573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9022.886307] LNet: Removed LNI 192.168.202.116@tcp [ 9023.221131] Key type .llcrypt unregistered [ 9023.222385] Key type ._llcrypt unregistered [ 9032.149911] Key type ._llcrypt registered [ 9032.151307] Key type .llcrypt registered [ 9032.191581] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 9039.091408] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 9039.469192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9039.514807] alg: No test for adler32 (adler32-zlib) [ 9040.376512] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 9040.469568] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 9042.047155] Key type lgssc registered [ 9042.409432] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9057.757516] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 9061.195179] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9061.200280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9062.284792] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9062.299259] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9062.332739] Lustre: lustre-MDT0000: new disk, initializing [ 9062.353503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9062.360667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9063.748843] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9068.004703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9068.047563] Lustre: 252607:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9068.069865] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9068.072103] Lustre: Skipped 1 previous similar message [ 9068.106630] Lustre: lustre-MDT0001: new disk, initializing [ 9068.142072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9068.156670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9068.160142] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9069.536933] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9071.830896] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9073.929382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9074.020263] Lustre: lustre-OST0000: new disk, initializing [ 9074.022310] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9074.025639] Lustre: 254238:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9074.053445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9076.030261] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9080.094567] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9080.149332] Lustre: lustre-OST0001: new disk, initializing [ 9080.151258] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9080.153658] Lustre: 255097:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9080.187247] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9080.732250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9080.737667] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9080.793753] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9080.803956] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9082.085088] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9087.585158] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9091.251354] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9094.111896] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9094.114650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9094.118090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9096.162827] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9096.164058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9096.166498] Lustre: Skipped 2 previous similar messages [ 9096.168915] Lustre: Skipped 2 previous similar messages [ 9100.263364] Lustre: server umount lustre-MDT0000 complete [ 9101.494261] LustreError: 252599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788878541 with bad export cookie 13064602867948429922 [ 9101.497742] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9101.498095] LustreError: 252599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9101.607653] Lustre: server umount lustre-MDT0001 complete [ 9113.607845] Lustre: server umount lustre-OST0000 complete [ 9125.212314] Lustre: server umount lustre-OST0001 complete [ 9129.724465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9132.847796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9135.704137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9138.733044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9145.315767] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 9148.878856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9148.886047] Lustre: lustre-MDT0000: reset Object Index mappings [ 9149.019890] LustreError: 260106:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9149.027312] LustreError: 260106:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9149.036515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9150.424820] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9153.135709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9153.141299] Lustre: lustre-MDT0001: reset Object Index mappings [ 9154.562573] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9155.547679] Lustre: 261248:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9157.874784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9157.991315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9157.993188] Lustre: Skipped 1 previous similar message [ 9159.012142] LustreError: 261787:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9159.014624] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9159.851281] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9162.627473] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9163.756237] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9164.792955] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9168.015560] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9169.214945] Lustre: 263114:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9173.988213] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9173.991183] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9173.992619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9173.992662] Lustre: Skipped 3 previous similar messages [ 9176.604906] Lustre: server umount lustre-MDT0000 complete [ 9177.881622] LustreError: 260089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788878618 with bad export cookie 13064602867948436600 [ 9177.884813] LustreError: 260089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9177.885157] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9178.006739] Lustre: server umount lustre-MDT0001 complete [ 9189.511895] Lustre: server umount lustre-OST0000 complete [ 9201.053595] Lustre: server umount lustre-OST0001 complete [ 9207.582467] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 10:44:06 (1788878646) [ 9213.348093] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 9217.020535] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 9229.358409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9229.441531] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9229.452644] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9229.491454] Lustre: lustre-MDT0000: new disk, initializing [ 9229.516171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9229.517947] Lustre: Skipped 1 previous similar message [ 9229.523647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9229.526534] Lustre: Skipped 1 previous similar message [ 9230.890028] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9235.009563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9235.055528] Lustre: 267366:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9235.175070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9235.178558] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9235.180532] Lustre: Skipped 1 previous similar message [ 9236.453761] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9240.102620] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9241.947288] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9244.294052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9244.395135] Lustre: lustre-OST0000: new disk, initializing [ 9244.396489] Lustre: Skipped 1 previous similar message [ 9244.398091] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9244.400966] Lustre: Skipped 2 previous similar messages [ 9244.403627] Lustre: 269337:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9245.814320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9245.816993] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9245.857030] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9246.408085] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9250.145384] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9251.714048] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9251.786541] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9253.283749] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9253.364497] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9256.150984] Lustre: server umount lustre-OST0000 complete [ 9261.023802] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9261.025940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9261.029124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9261.030584] Lustre: Skipped 3 previous similar messages [ 9266.655926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9266.658057] Lustre: Skipped 2 previous similar messages [ 9267.710991] Lustre: server umount lustre-MDT0000 complete [ 9269.034064] LustreError: 268164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788878709 with bad export cookie 13064602867948438672 [ 9269.037109] LustreError: 268164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9269.037146] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9279.510918] Lustre: server umount lustre-MDT0001 complete [ 9286.083646] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 9289.610270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9289.747933] LustreError: 272265:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9289.761747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9289.764209] Lustre: Skipped 2 previous similar messages [ 9291.130038] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9294.145754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9295.739657] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9296.835245] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9299.375585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9301.752710] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9305.153401] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9305.215750] Lustre: lustre-OST0001: new disk, initializing [ 9305.217598] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9305.221254] Lustre: 274574:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9307.015670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9307.019038] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9307.056101] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9307.466266] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9312.126463] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9313.398975] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9315.127730] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9315.135752] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9317.345886] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9317.348247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9317.349952] Lustre: Skipped 4 previous similar messages [ 9317.352026] Lustre: Skipped 2 previous similar messages [ 9323.166165] Lustre: server umount lustre-MDT0000 complete [ 9324.450344] LustreError: 272247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788878764 with bad export cookie 13064602867948439554 [ 9324.453803] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9324.455387] LustreError: 272247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9324.569912] Lustre: server umount lustre-MDT0001 complete [ 9335.810881] Lustre: server umount lustre-OST0000 complete [ 9347.421924] Lustre: server umount lustre-OST0001 complete [ 9353.336725] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 9357.092182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9357.236730] LustreError: 277620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9357.254061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9357.255601] Lustre: Skipped 3 previous similar messages [ 9358.767975] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9361.855710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9363.521455] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9364.517309] Lustre: 278762:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9366.864182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9368.034974] LustreError: 279114:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9368.043841] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9369.049707] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9372.213311] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9374.253206] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9377.553891] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9377.774996] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9378.884699] Lustre: 280634:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9380.789018] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9380.824031] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9384.145712] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 10:47:02 (1788878822) [ 9397.728790] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9397.731702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9397.736500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9397.738275] Lustre: Skipped 4 previous similar messages [ 9401.821152] Lustre: server umount lustre-MDT0000 complete [ 9403.140455] LustreError: 278364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788878843 with bad export cookie 13064602867948445861 [ 9403.144016] LustreError: 278364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9403.144091] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9403.360076] LustreError: 277616:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9403.367344] LustreError: 277616:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9409.326801] Lustre: server umount lustre-MDT0001 complete [ 9416.574065] Lustre: server umount lustre-OST0000 complete [ 9423.955272] Lustre: server umount lustre-OST0001 complete [ 9426.019163] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 9429.392496] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 9444.416208] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 9448.000293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9448.076874] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9448.085709] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9448.119474] Lustre: lustre-MDT0000: new disk, initializing [ 9448.144365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9449.496268] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9453.626744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9453.679540] Lustre: 284863:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9453.792281] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9455.171153] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9459.626489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9459.714889] Lustre: 286496:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9461.214049] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9461.640119] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9465.644544] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9467.114569] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9467.675726] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9471.985639] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9477.074993] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 10:48:35 (1788878915) [ 9479.647762] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9479.649886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9479.653359] Lustre: Skipped 5 previous similar messages [ 9479.655170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9479.657053] Lustre: Skipped 8 previous similar messages [ 9483.928957] Lustre: server umount lustre-MDT0000 complete [ 9485.080850] LustreError: 287368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788878925 with bad export cookie 13064602867948453750 [ 9485.083329] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9485.084281] LustreError: 287368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9485.196849] Lustre: server umount lustre-MDT0001 complete [ 9496.065105] Lustre: server umount lustre-OST0000 complete [ 9507.610042] Lustre: server umount lustre-OST0001 complete [ 9510.477783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9510.626106] LustreError: 289608:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9510.631297] LustreError: 289608:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9510.652915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9510.655911] Lustre: Skipped 7 previous similar messages [ 9511.931497] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9514.730428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9516.132820] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9518.557536] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9520.498078] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9522.777158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9524.799595] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9527.177936] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9542.677366] LustreError: 291438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9542.682141] LustreError: 291438:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 9543.454303] loop1: detected capacity change from 0 to 204800000 [ 9543.532400] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9543.537905] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9543.541756] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9545.279771] loop1: detected capacity change from 0 to 204800000 [ 9547.154156] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9547.189844] LustreError: 289598:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs25836: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 9547.194398] LustreError: 289598:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs25836 [ 9547.196073] LustreError: 289598:0:(mgs_handler.c:517:mgs_target_reg()) fs25836-MDT0001: Error checking target index: rc = -13 [ 9547.198728] LustreError: fs25836-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 9547.201665] LustreError: 293348:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [ 9547.204099] LustreError: 293348:0:(tgt_mount.c:2132:server_put_super()) no obd fs25836-MDT0001 [ 9547.206672] LustreError: 293348:0:(tgt_mount.c:116:server_deregister_mount()) fs25836-MDT0001 not registered [ 9547.218624] Lustre: server umount fs25836-MDT0001 complete [ 9547.220070] LustreError: 293348:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 9550.092087] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9550.118243] Lustre: Setting parameter fs25836-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs25836-MDT0001 [ 9550.123215] Lustre: Skipped 2 previous similar messages [ 9550.137198] Lustre: srv-fs25836-MDT0001: No data found on store. Initialize space. [ 9550.138807] Lustre: Skipped 4 previous similar messages [ 9550.177170] Lustre: fs25836-MDT0001: new disk, initializing [ 9550.179031] Lustre: Skipped 3 previous similar messages [ 9550.210439] Lustre: srv-fs25836-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 9550.213056] Lustre: 293814:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs25836-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9550.217521] LustreError: 293814:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs25836-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9551.525592] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9554.428137] Lustre: server umount fs25836-MDT0001 complete [ 9558.526895] loop1: detected capacity change from 0 to 204800000 [ 9560.354892] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9560.456102] Lustre: 295378:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs25836-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9560.458513] LustreError: 295378:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs25836-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9561.666047] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9563.178225] Lustre: server umount fs25836-MDT0001 complete [ 9567.201775] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9567.203113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9567.206705] Lustre: Skipped 5 previous similar messages [ 9567.208433] Lustre: Skipped 5 previous similar messages [ 9572.439161] Lustre: server umount lustre-MDT0000 complete [ 9575.181389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9575.217448] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9576.759783] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9579.502366] loop1: detected capacity change from 0 to 204800000 [ 9580.517381] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9581.465905] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9581.592374] Lustre: 297819:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs25836-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9581.594727] LustreError: 297819:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs25836-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9583.061897] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9584.764628] Lustre: server umount fs25836-MDT0001 complete [ 9591.776140] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9597.530190] Lustre: server umount lustre-OST0000 complete [ 9613.279096] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9613.344984] Lustre: server umount lustre-MDT0000 complete [ 9614.564314] LustreError: 289589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788879054 with bad export cookie 13064602867948456424 [ 9614.567464] LustreError: 289589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9614.568268] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9614.677564] Lustre: server umount lustre-MDT0001 complete [ 9618.342099] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 9619.404661] Key type lgssc unregistered [ 9619.534320] LNet: 300157:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9619.536404] LNetError: 300157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9619.549334] LNet: Removed LNI 192.168.202.116@tcp [ 9619.899098] Key type .llcrypt unregistered [ 9619.900101] Key type ._llcrypt unregistered [ 9627.244401] Key type ._llcrypt registered [ 9627.245162] Key type .llcrypt registered [ 9627.279870] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 9627.638926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9627.652164] alg: No test for adler32 (adler32-zlib) [ 9628.502234] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 9628.595859] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 9630.175271] Key type lgssc registered [ 9630.515697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9633.139869] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9633.146639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9634.308567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9635.744289] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9638.626857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9638.769449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9640.182590] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9642.884440] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9644.943587] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9647.145960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9647.280438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9649.240787] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9649.316342] LustreError: 303607:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9649.322260] LustreError: 303607:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9651.828118] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9654.755388] LustreError: 303228:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9658.849752] LustreError: 303228:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9658.853697] LustreError: 303228:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9662.490833] LustreError: 303228:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9667.605993] LustreError: 303228:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9667.610774] LustreError: 303228:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9668.686303] loop1: detected capacity change from 0 to 204800000 [ 9668.782025] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9668.786712] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9668.791308] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9670.432698] loop1: detected capacity change from 0 to 204800000 [ 9672.280875] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9672.333817] Lustre: fs2636-OST0000: new disk, initializing [ 9672.335995] Lustre: srv-fs2636-OST0000: No data found on store. Initialize space. [ 9672.338465] Lustre: 305154:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9672.363110] Lustre: fs2636-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9674.437542] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9677.846535] LustreError: 303900:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9677.853804] LustreError: 303900:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9678.111440] Lustre: server umount fs2636-OST0000 complete [ 9682.444828] loop1: detected capacity change from 0 to 204800000 [ 9684.507994] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9684.606064] Lustre: fs2636-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9686.758798] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9689.564673] loop2: detected capacity change from 0 to 204800000 [ 9689.633199] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9689.637264] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9689.640603] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9691.407928] loop2: detected capacity change from 0 to 204800000 [ 9693.352974] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9693.387364] LustreError: fs2636-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 9693.391889] LustreError: fs2636-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 9693.394914] LustreError: 308189:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [ 9693.397282] LustreError: 308189:0:(tgt_mount.c:2132:server_put_super()) no obd fs2636-OST0001 [ 9693.399297] LustreError: 308189:0:(tgt_mount.c:116:server_deregister_mount()) fs2636-OST0001 not registered [ 9693.410414] Lustre: server umount fs2636-OST0001 complete [ 9693.412080] LustreError: 308189:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 9694.690373] LustreError: 303607:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9694.697387] LustreError: 303607:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 9696.842294] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9696.886550] Lustre: fs2636-OST0001: new disk, initializing [ 9696.888960] Lustre: srv-fs2636-OST0001: No data found on store. Initialize space. [ 9696.892503] Lustre: 308739:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9696.921702] Lustre: fs2636-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9698.982184] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9702.531367] Lustre: server umount fs2636-OST0000 complete [ 9705.742714] Lustre: server umount fs2636-OST0001 complete [ 9710.058888] loop1: detected capacity change from 0 to 204800000 [ 9712.042970] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9714.086712] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9716.370102] loop2: detected capacity change from 0 to 204800000 [ 9718.299172] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9718.367929] Lustre: fs2636-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9718.370158] Lustre: Skipped 1 previous similar message [ 9720.248255] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9723.054776] Lustre: server umount fs2636-OST0000 complete [ 9726.071322] Lustre: server umount fs2636-OST0001 complete [ 9729.046824] LustreError: 303228:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9729.055332] LustreError: 303228:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [ 9730.530059] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9730.531360] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9730.534465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9730.537306] Lustre: Skipped 1 previous similar message [ 9735.649111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9735.650994] Lustre: Skipped 1 previous similar message [ 9740.768636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9740.770780] Lustre: Skipped 1 previous similar message [ 9740.771986] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 9744.351099] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9744.406920] Lustre: server umount lustre-OST0000 complete [ 9745.888469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9745.889155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9745.894142] Lustre: Skipped 2 previous similar messages [ 9751.008057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9751.009988] Lustre: Skipped 2 previous similar messages [ 9751.698871] Lustre: server umount lustre-MDT0000 complete [ 9752.840919] LustreError: 301382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788879193 with bad export cookie 12648036016063277234 [ 9752.844377] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9752.845637] LustreError: 301382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9752.962757] Lustre: server umount lustre-MDT0001 complete [ 9756.332347] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 9757.241124] Key type lgssc unregistered [ 9757.354309] LNet: 314650:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9757.356890] LNetError: 314650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9757.365288] LNet: Removed LNI 192.168.202.116@tcp [ 9757.656131] Key type .llcrypt unregistered [ 9757.657560] Key type ._llcrypt unregistered [ 9767.824204] Key type ._llcrypt registered [ 9767.825506] Key type .llcrypt registered [ 9767.871549] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 9773.993419] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 9774.292052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9774.301118] alg: No test for adler32 (adler32-zlib) [ 9775.148255] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 9775.230585] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 9776.807149] Key type lgssc registered [ 9777.158871] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9788.747867] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9788.753184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9789.835605] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9789.846122] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9789.886462] Lustre: lustre-MDT0000: new disk, initializing [ 9789.912340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9789.919524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9791.280944] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9795.387907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9795.428403] Lustre: 318473:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9795.451712] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9795.453399] Lustre: Skipped 1 previous similar message [ 9795.490177] Lustre: lustre-MDT0001: new disk, initializing [ 9795.523399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9795.539524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9795.542993] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9796.844903] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9800.413437] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9802.189584] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9804.440890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9804.539878] Lustre: lustre-OST0000: new disk, initializing [ 9804.541355] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9804.543444] Lustre: 320441:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9804.568621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9806.346515] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9807.029109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9807.032486] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9807.060953] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9809.979782] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9810.633112] Lustre: Mounted lustre-client - version 2.17.57_113_gf5ca795 [ 9810.958730] Lustre: Unmounted lustre-client [ 9816.034353] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9816.035139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9816.039888] Lustre: Skipped 1 previous similar message [ 9819.853771] Lustre: server umount lustre-OST0000 complete [ 9821.152513] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9821.152549] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9821.153062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9821.153066] Lustre: Skipped 1 previous similar message [ 9821.159697] Lustre: Skipped 1 previous similar message [ 9826.272032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9826.273457] Lustre: Skipped 2 previous similar messages [ 9827.156991] Lustre: server umount lustre-MDT0000 complete [ 9828.270101] LustreError: 318467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788879268 with bad export cookie 11607333365295251765 [ 9828.273385] LustreError: 318467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9828.273574] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9828.381682] Lustre: server umount lustre-MDT0001 complete [ 9831.911572] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 9832.905600] Key type lgssc unregistered [ 9867.871349] LNet: 322597:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9867.874460] LNetError: 322597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9942.629285] LNet: Removed LNI 192.168.202.116@tcp [ 9942.937867] Key type .llcrypt unregistered [ 9942.939458] Key type ._llcrypt unregistered [ 9948.641382] Key type ._llcrypt registered [ 9948.642272] Key type .llcrypt registered [ 9948.682721] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 9955.074478] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 9955.379616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9955.406869] alg: No test for adler32 (adler32-zlib) [ 9956.256684] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 9956.342290] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 9957.927123] Key type lgssc registered [ 9958.273773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9967.653228] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 10:56:46 (1788879406) [ 9969.735064] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9969.738882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9970.813749] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9970.823115] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9970.855051] Lustre: lustre-MDT0000: new disk, initializing [ 9970.873787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9970.880408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9972.121369] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9976.135103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9976.171995] Lustre: 325792:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9976.195034] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9976.196467] Lustre: Skipped 1 previous similar message [ 9976.228863] Lustre: lustre-MDT0001: new disk, initializing [ 9976.257768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9976.268886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9976.272696] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9977.525410] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9981.038737] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9982.811315] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9985.034838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9985.134300] Lustre: lustre-OST0000: new disk, initializing [ 9985.135976] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9985.138710] Lustre: 327757:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9985.163521] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9986.295735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9986.298735] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9986.327681] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9986.954921] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9990.534138] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9996.770346] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9996.770463] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9996.775386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9996.776911] Lustre: Skipped 1 previous similar message [10001.891817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10001.893763] Lustre: Skipped 1 previous similar message [10002.134902] Lustre: server umount lustre-OST0000 complete [10007.011313] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10007.011685] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10007.014966] Lustre: Skipped 1 previous similar message [10007.017983] LustreError: Skipped 1 previous similar message [10007.019972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10007.023779] Lustre: Skipped 1 previous similar message [10009.559841] Lustre: server umount lustre-MDT0000 complete [10010.787271] LustreError: 325784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788879451 with bad export cookie 14370103709666119770 [10010.789781] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10010.792401] LustreError: 325784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10010.901404] Lustre: server umount lustre-MDT0001 complete [10014.388310] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [10015.336841] Key type lgssc unregistered [10015.452321] LNet: 329665:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10015.454125] LNetError: 329665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10015.465483] LNet: Removed LNI 192.168.202.116@tcp [10015.760657] Key type .llcrypt unregistered [10015.762096] Key type ._llcrypt unregistered [10023.542778] Key type ._llcrypt registered [10023.544319] Key type .llcrypt registered [10023.582635] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [10023.930238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10023.983809] alg: No test for adler32 (adler32-zlib) [10024.833821] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [10024.929508] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [10026.511105] Key type lgssc registered [10026.843341] Lustre: Echo OBD driver; http://www.lustre.org/ [10029.367981] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10029.373404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10030.513345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10031.796085] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10034.525373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10034.644742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10035.886628] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10038.199269] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10039.974073] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10042.135114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10042.239193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10044.136272] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10046.616466] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10060.533606] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10061.032150] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10062.361359] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.16@tcp (stopping) [10064.339952] Lustre: server umount lustre-MDT0000 complete [10065.375992] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10065.378801] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10065.383742] LustreError: 331638:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10065.387625] LustreError: 331638:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10065.548638] LustreError: 330893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788879505 with bad export cookie 10997818753507721264 [10065.552531] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10065.555567] LustreError: 330893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10065.888234] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10065.888905] LustreError: 333245:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10065.893182] Lustre: Skipped 1 previous similar message [10065.903501] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10067.476081] LustreError: 332351:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10067.482894] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.16@tcp (stopping) [10071.007763] LustreError: 330905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10071.008079] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10071.715339] Lustre: server umount lustre-MDT0001 complete [10087.391230] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10087.442089] Lustre: server umount lustre-OST0000 complete [10090.189386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10091.441553] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10092.291412] Lustre: 335107:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.116@tcp [10092.946514] Lustre: server umount MGS complete [10095.624859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10095.771946] LustreError: 335678:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10095.793606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10097.085109] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10099.708697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10099.816646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10101.101944] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10103.699793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10105.660626] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10107.943034] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10108.415945] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [10110.056501] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10118.680544] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 8 sec [10120.160476] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10120.162552] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10120.167545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10120.169455] Lustre: Skipped 1 previous similar message [10125.832772] Lustre: server umount lustre-OST0000 complete [10130.400852] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10130.401363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10130.404921] Lustre: Skipped 2 previous similar messages [10130.405183] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10130.406611] Lustre: Skipped 3 previous similar messages [10133.145993] Lustre: server umount lustre-MDT0000 complete [10134.354990] LustreError: 335659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788879574 with bad export cookie 10997818753507722090 [10134.358372] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10134.360245] LustreError: 335659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10134.470246] Lustre: server umount lustre-MDT0001 complete [10137.830791] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [10138.735077] Key type lgssc unregistered [10138.858330] LNet: 339109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10138.861024] LNetError: 339109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10138.868243] LNet: Removed LNI 192.168.202.116@tcp [10139.168460] Key type .llcrypt unregistered [10139.169474] Key type ._llcrypt unregistered [10144.847713] Key type ._llcrypt registered [10144.848719] Key type .llcrypt registered [10144.883774] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [10151.103237] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [10151.475366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10151.498667] alg: No test for adler32 (adler32-zlib) [10152.341443] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [10152.422895] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [10153.999153] Key type lgssc registered [10154.376127] Lustre: Echo OBD driver; http://www.lustre.org/ [10165.401967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10165.406639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10166.479933] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10166.489252] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10166.522890] Lustre: lustre-MDT0000: new disk, initializing [10166.542780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10166.549072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10167.828920] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10171.865996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10171.907051] Lustre: 342238:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10171.927972] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10171.929690] Lustre: Skipped 1 previous similar message [10171.962637] Lustre: lustre-MDT0001: new disk, initializing [10171.993857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10172.005625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10172.009405] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10173.368867] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10176.985816] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10178.737256] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10180.940182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10181.033183] Lustre: lustre-OST0000: new disk, initializing [10181.035142] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10181.037714] Lustre: 344203:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10181.062176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10182.895319] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10186.574075] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10187.905994] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10188.789759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10188.793648] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10188.821566] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10188.991734] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [10190.345328] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10190.413808] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10193.890132] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10193.891265] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10193.891981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10193.892927] LustreError: Skipped 1 previous similar message [10193.896751] Lustre: Skipped 1 previous similar message [10197.138544] Lustre: server umount lustre-OST0000 complete [10199.008478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10199.009349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10199.013377] Lustre: Skipped 1 previous similar message [10199.016693] Lustre: Skipped 2 previous similar messages [10204.128444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10204.129915] Lustre: Skipped 1 previous similar message [10204.570936] Lustre: server umount lustre-MDT0000 complete [10205.838130] LustreError: 343030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788879646 with bad export cookie 5072049458360849559 [10205.840693] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10205.842303] LustreError: 343030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10205.968552] Lustre: server umount lustre-MDT0001 complete [10208.961848] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 11:00:47 (1788879647) [10214.059763] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [10217.251559] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [10228.520161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10228.615181] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10228.627273] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10228.661751] Lustre: lustre-MDT0000: new disk, initializing [10228.683857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10228.690055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10230.001184] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10234.001761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10234.045136] Lustre: 348968:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10234.065099] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10234.066385] Lustre: Skipped 1 previous similar message [10234.095097] Lustre: lustre-MDT0001: new disk, initializing [10234.122651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10234.134281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10234.138621] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10235.390518] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10238.976495] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10240.768700] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10242.909433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10243.007274] Lustre: lustre-OST0000: new disk, initializing [10243.008903] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10243.011459] Lustre: 350939:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10243.040120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10244.927886] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10248.697785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10248.701263] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10248.734565] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10248.735128] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10258.914306] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10258.914889] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10258.918154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10258.919557] Lustre: Skipped 1 previous similar message [10258.922213] LustreError: Skipped 1 previous similar message [10260.507131] Lustre: server umount lustre-OST0000 complete [10264.032594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10264.033244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10264.036931] Lustre: Skipped 1 previous similar message [10264.039160] Lustre: Skipped 1 previous similar message [10274.272036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10274.274236] Lustre: Skipped 3 previous similar messages [10276.319094] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10276.379505] Lustre: server umount lustre-MDT0000 complete [10277.644230] LustreError: 348962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788879718 with bad export cookie 5072049458360850427 [10277.646482] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10277.648580] LustreError: 348962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10277.763871] Lustre: server umount lustre-MDT0001 complete [10281.378396] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [10282.382682] Key type lgssc unregistered [10282.514510] LNet: 353086:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10282.520882] LNetError: 353086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10282.530431] LNet: Removed LNI 192.168.202.116@tcp [10282.847903] Key type .llcrypt unregistered [10282.849174] Key type ._llcrypt unregistered [10315.852166] Key type ._llcrypt registered [10315.852990] Key type .llcrypt registered [10315.888608] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [10315.987215] Key type .llcrypt unregistered [10315.988222] Key type ._llcrypt unregistered [10343.393424] Key type ._llcrypt registered [10343.394366] Key type .llcrypt registered [10343.432283] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [10343.531217] Key type .llcrypt unregistered [10343.532710] Key type ._llcrypt unregistered [10355.051067] Key type ._llcrypt registered [10355.052169] Key type .llcrypt registered [10355.091681] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [10355.209209] Key type .llcrypt unregistered [10355.210654] Key type ._llcrypt unregistered [10367.306757] Key type ._llcrypt registered [10367.307874] Key type .llcrypt registered [10367.345549] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [10367.453373] Key type .llcrypt unregistered [10367.454811] Key type ._llcrypt unregistered [10395.846643] Key type ._llcrypt registered [10395.847885] Key type .llcrypt registered [10395.884234] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [10395.980234] Key type .llcrypt unregistered [10395.981198] Key type ._llcrypt unregistered [10406.908499] Key type ._llcrypt registered [10406.909309] Key type .llcrypt registered [10406.942350] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [10407.039450] Key type .llcrypt unregistered [10407.040567] Key type ._llcrypt unregistered [10417.848725] Key type ._llcrypt registered [10417.849685] Key type .llcrypt registered [10417.881881] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [10417.964323] Key type .llcrypt unregistered [10417.965232] Key type ._llcrypt unregistered [10428.999512] Key type ._llcrypt registered [10429.000610] Key type .llcrypt registered [10429.033635] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [10429.137326] Key type .llcrypt unregistered [10429.138190] Key type ._llcrypt unregistered [10441.922744] Key type ._llcrypt registered [10441.923614] Key type .llcrypt registered [10441.957933] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [10442.052380] Key type .llcrypt unregistered [10442.053488] Key type ._llcrypt unregistered [10453.662379] Key type ._llcrypt registered [10453.663362] Key type .llcrypt registered [10453.696561] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [10453.793390] Key type .llcrypt unregistered [10453.794405] Key type ._llcrypt unregistered [10474.653433] Key type ._llcrypt registered [10474.654565] Key type .llcrypt registered [10474.693937] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [10474.794211] Key type .llcrypt unregistered [10474.795478] Key type ._llcrypt unregistered [10485.655709] Key type ._llcrypt registered [10485.656743] Key type .llcrypt registered [10485.694272] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [10485.795437] Key type .llcrypt unregistered [10485.796789] Key type ._llcrypt unregistered [10496.247442] Key type ._llcrypt registered [10496.248235] Key type .llcrypt registered [10496.282457] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [10496.362286] Key type .llcrypt unregistered [10496.363111] Key type ._llcrypt unregistered [10514.075252] Key type ._llcrypt registered [10514.076321] Key type .llcrypt registered [10514.113301] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [10520.188570] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [10520.512848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10520.556460] alg: No test for adler32 (adler32-zlib) [10521.410136] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [10521.493960] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [10523.079425] Key type lgssc registered [10523.450955] Lustre: Echo OBD driver; http://www.lustre.org/ [10534.289026] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10534.294136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10535.368950] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [10535.379460] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [10535.415287] Lustre: tst250fs-MDT0000: new disk, initializing [10535.434879] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10535.439831] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10536.768695] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10540.593661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10540.630886] Lustre: 372930:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10540.650201] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [10540.651571] Lustre: Skipped 1 previous similar message [10540.681414] Lustre: tst250fs-MDT0001: new disk, initializing [10540.707723] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10540.717736] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10540.720497] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10541.898737] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10544.110875] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10546.367669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10546.455112] Lustre: tst250fs-OST0000: new disk, initializing [10546.457394] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [10546.461393] Lustre: 374562:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10546.494530] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10548.336640] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10548.981131] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10548.983408] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10549.022951] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10552.246673] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10552.302330] Lustre: tst250fs-OST0001: new disk, initializing [10552.303660] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [10552.305908] Lustre: 375433:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10552.333941] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10554.083245] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10556.292846] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10556.297721] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10556.334153] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10556.567917] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [10566.626504] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10566.628036] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [10566.630482] Lustre: Skipped 3 previous similar messages [10566.632223] Lustre: Skipped 2 previous similar messages [10566.635288] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10571.745989] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [10571.747889] Lustre: Skipped 4 previous similar messages [10572.564602] Lustre: server umount tst250fs-MDT0000 complete [10573.658151] LustreError: 374193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788880014 with bad export cookie 8625511655819453063 [10573.661741] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10573.661893] LustreError: 374193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10573.772035] Lustre: server umount tst250fs-MDT0001 complete [10585.091128] Lustre: server umount tst250fs-OST0000 complete [10596.575621] Lustre: server umount tst250fs-OST0001 complete [10602.565124] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [10605.386030] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [10615.630832] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 11:07:34 (1788880054) [10616.191602] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [10621.614948] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [10625.049420] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [10635.800101] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10451 sec ========= 11:07:54 (1788880074) [10636.443200] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 11:07:55 (1788880075) === [10637.680959] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 11:07:56 (1788880076) === [10650.960427] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [10652.035766] Key type lgssc unregistered [10652.153376] LNet: 384539:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10652.155649] LNetError: 384539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10652.164325] LNet: Removed LNI 192.168.202.116@tcp [10652.482115] Key type .llcrypt unregistered [10652.483414] Key type ._llcrypt unregistered