[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 557359643 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002354] x2apic enabled [ 0.003013] Switched APIC routing to physical x2apic. [ 0.005020] kvm-guest: setup PV IPIs [ 0.009533] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.010000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010032] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011017] pid_max: default: 32768 minimum: 301 [ 0.012166] LSM: Security Framework initializing [ 0.013071] Yama: becoming mindful. [ 0.014044] SELinux: Initializing. [ 0.015093] *** VALIDATE selinux *** [ 0.025795] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.031785] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.032167] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033124] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034153] *** VALIDATE tmpfs *** [ 0.037102] *** VALIDATE proc *** [ 0.038034] *** VALIDATE cgroup *** [ 0.039017] *** VALIDATE cgroup2 *** [ 0.040045] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.041183] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.042013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.043039] Spectre V2 : User space: Vulnerable [ 0.044015] Speculative Store Bypass: Vulnerable [ 0.048587] debug: unmapping init [mem 0xffffffffab459000-0xffffffffab460fff] [ 0.051726] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.053331] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.054032] ... version: 2 [ 0.055026] ... bit width: 48 [ 0.056022] ... generic registers: 4 [ 0.057025] ... value mask: 0000ffffffffffff [ 0.058025] ... max period: 00007fffffffffff [ 0.059023] ... fixed-purpose events: 3 [ 0.060022] ... event mask: 000000070000000f [ 0.061782] rcu: Hierarchical SRCU implementation. [ 0.065188] smp: Bringing up secondary CPUs ... [ 0.067720] x86: Booting SMP configuration: [ 0.068028] .... node #0, CPUs: #1 #2 #3 [ 0.087177] smp: Brought up 1 node, 4 CPUs [ 0.089016] smpboot: Max logical packages: 1 [ 0.090011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.162031] node 0 deferred pages initialised in 61ms [ 0.171293] devtmpfs: initialized [ 0.173301] x86/mm: Memory block size: 128MB [ 0.177000] gcov: version magic: 0x41383552 [ 0.179400] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.183206] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.189327] pinctrl core: initialized pinctrl subsystem [ 0.192244] [ 0.193011] ************************************************************* [ 0.195018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.199015] ** ** [ 0.204019] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.210017] ** ** [ 0.213013] ** This means that this kernel is built to expose internal ** [ 0.225019] ** IOMMU data structures, which may compromise security on ** [ 0.233019] ** your system. ** [ 0.241024] ** ** [ 0.248015] ** If you see this message and you are not debugging the ** [ 0.256017] ** kernel, report this immediately to your vendor! ** [ 0.264025] ** ** [ 0.268024] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.272015] ************************************************************* [ 0.296260] NET: Registered protocol family 16 [ 0.299465] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.303457] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.310076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.322507] cpuidle: using governor menu [ 0.324969] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.326169] PCI: Using configuration type 1 for base access [ 0.327152] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.343088] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.345054] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.351113] cryptd: max_cpu_qlen set to 1000 [ 0.358904] ACPI: Added _OSI(Module Device) [ 0.364045] ACPI: Added _OSI(Processor Device) [ 0.367032] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.371018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.388645] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.406000] ACPI: Interpreter enabled [ 0.409080] ACPI: PM: (supports S0 S3 S4 S5) [ 0.412027] ACPI: Using IOAPIC for interrupt routing [ 0.419077] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.430434] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.454331] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.457052] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.472022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.481094] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.493299] acpiphp: Slot [2] registered [ 0.496318] acpiphp: Slot [5] registered [ 0.498622] acpiphp: Slot [6] registered [ 0.501238] acpiphp: Slot [7] registered [ 0.502144] acpiphp: Slot [8] registered [ 0.506250] acpiphp: Slot [9] registered [ 0.508248] acpiphp: Slot [10] registered [ 0.510265] acpiphp: Slot [3] registered [ 0.512178] acpiphp: Slot [4] registered [ 0.514128] acpiphp: Slot [11] registered [ 0.516223] acpiphp: Slot [12] registered [ 0.518487] acpiphp: Slot [13] registered [ 0.523328] acpiphp: Slot [14] registered [ 0.525248] acpiphp: Slot [15] registered [ 0.527306] acpiphp: Slot [16] registered [ 0.529132] acpiphp: Slot [17] registered [ 0.532362] acpiphp: Slot [18] registered [ 0.533126] acpiphp: Slot [19] registered [ 0.541134] acpiphp: Slot [20] registered [ 0.543166] acpiphp: Slot [21] registered [ 0.546452] acpiphp: Slot [22] registered [ 0.548261] acpiphp: Slot [23] registered [ 0.551241] acpiphp: Slot [24] registered [ 0.554217] acpiphp: Slot [25] registered [ 0.556208] acpiphp: Slot [26] registered [ 0.559589] acpiphp: Slot [27] registered [ 0.561155] acpiphp: Slot [28] registered [ 0.567165] acpiphp: Slot [29] registered [ 0.569372] acpiphp: Slot [30] registered [ 0.570140] acpiphp: Slot [31] registered [ 0.576111] PCI host bridge to bus 0000:00 [ 0.581030] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.587030] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.596027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.605031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.606034] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.615029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.622268] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.635686] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.641346] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.660024] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.675083] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.681022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.683021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.696000] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.704349] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.712213] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.717048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.722046] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.734018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.768019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.779019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.796590] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.810018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.824025] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.859022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.874162] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.880020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.891019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.915018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.944062] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.953018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.971030] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 1.014021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 1.036000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 1.045021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 1.066020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 1.091000] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 1.106248] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 1.119021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 1.128019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.153020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 1.169000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.182066] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.195026] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.216020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.237328] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.246554] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.251314] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.261326] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.266267] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.277043] iommu: Default domain type: Passthrough [ 1.282255] SCSI subsystem initialized [ 1.285389] ACPI: bus type USB registered [ 1.288617] usbcore: registered new interface driver usbfs [ 1.293179] usbcore: registered new interface driver hub [ 1.297110] usbcore: registered new device driver usb [ 1.300203] pps_core: LinuxPPS API ver. 1 registered [ 1.306015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.311510] PTP clock support registered [ 1.315110] EDAC MC: Ver: 3.0.0 [ 1.321018] PCI: Using ACPI for IRQ routing [ 1.324968] NetLabel: Initializing [ 1.327014] NetLabel: domain hash size = 128 [ 1.329016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.333099] NetLabel: unlabeled traffic allowed by default [ 1.336109] vgaarb: loaded [ 1.338665] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.339017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.342761] clocksource: Switched to clocksource kvm-clock [ 1.556786] VFS: Disk quotas dquot_6.6.0 [ 1.559025] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.564373] *** VALIDATE ramfs *** [ 1.568165] *** VALIDATE hugetlbfs *** [ 1.572151] pnp: PnP ACPI init [ 1.579749] pnp: PnP ACPI: found 6 devices [ 1.615085] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.618766] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.621336] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.623848] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.626643] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.631163] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.635238] NET: Registered protocol family 2 [ 1.638354] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.644498] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.648451] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.654316] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.657917] TCP: Hash tables configured (established 65536 bind 65536) [ 1.661587] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.665581] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.668976] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.672574] NET: Registered protocol family 1 [ 1.675559] RPC: Registered named UNIX socket transport module. [ 1.678417] RPC: Registered udp transport module. [ 1.680476] RPC: Registered tcp transport module. [ 1.682445] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.685677] NET: Registered protocol family 44 [ 1.688498] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.691552] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.694361] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.697885] PCI: CLS 0 bytes, default 64 [ 1.700808] Unpacking initramfs... [ 5.359133] debug: unmapping init [mem 0xffff89253cc54000-0xffff89253ffbffff] [ 5.371686] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 5.376699] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 5.382697] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 7.257762] Initialise system trusted keyrings [ 7.260682] Key type blacklist registered [ 7.265111] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 7.283976] zbud: loaded [ 7.300936] *** VALIDATE nfs *** [ 7.302609] *** VALIDATE nfs4 *** [ 7.306380] pstore: using deflate compression [ 7.332105] Platform Keyring initialized [ 7.613468] NET: Registered protocol family 38 [ 7.617148] Key type asymmetric registered [ 7.620116] Asymmetric key parser 'x509' registered [ 7.622716] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 7.634800] io scheduler mq-deadline registered [ 7.636670] io scheduler kyber registered [ 7.638899] io scheduler bfq registered [ 7.641747] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 7.647275] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 7.651463] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 7.655476] ACPI: Power Button [PWRF] [ 7.676514] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 7.702582] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 7.727158] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 7.743596] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 7.783610] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 7.831944] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 7.874418] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 7.883492] Non-volatile memory driver v1.3 [ 7.886942] Linux agpgart interface v0.103 [ 7.947505] virtio_blk virtio1: [vda] 145960 512-byte logical blocks (74.7 MB/71.3 MiB) [ 7.952511] vda: detected capacity change from 0 to 74731520 [ 7.993730] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 8.001329] vdb: detected capacity change from 0 to 1073741824 [ 8.043801] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 8.049826] vdc: detected capacity change from 0 to 2621440000 [ 8.098199] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 8.103158] vdd: detected capacity change from 0 to 2621440000 [ 8.133300] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.143772] vde: detected capacity change from 0 to 4294967296 [ 8.180821] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.186041] vdf: detected capacity change from 0 to 4294967296 [ 8.195576] libphy: Fixed MDIO Bus: probed [ 8.210186] usbcore: registered new interface driver usbserial_generic [ 8.213152] usbserial: USB Serial support registered for generic [ 8.218203] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 8.222645] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 8.224057] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 8.227234] mousedev: PS/2 mouse device common for all mice [ 8.234094] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 8.236268] rtc_cmos 00:05: RTC can wake from S4 [ 8.242630] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 8.245930] rtc_cmos 00:05: registered as rtc0 [ 8.248593] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 8.252781] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 8.252826] intel_pstate: CPU model not supported [ 8.262723] hid: raw HID events driver (C) Jiri Kosina [ 8.271299] usbcore: registered new interface driver usbhid [ 8.278510] usbhid: USB HID core driver [ 8.281847] drop_monitor: Initializing network drop monitor service [ 8.291144] Initializing XFRM netlink socket [ 8.294431] NET: Registered protocol family 10 [ 8.308770] Segment Routing with IPv6 [ 8.312818] NET: Registered protocol family 17 [ 8.319030] mpls_gso: MPLS GSO support [ 8.331770] RAS: Correctable Errors collector initialized. [ 8.335532] AVX version of gcm_enc/dec engaged. [ 8.343094] AES CTR mode by8 optimization enabled [ 8.484757] sched_clock: Marking stable (8484600013, 0)->(10851398991, -2366798978) [ 8.488394] registered taskstats version 1 [ 8.490699] Loading compiled-in X.509 certificates [ 8.492905] zswap: loaded using pool lzo/zbud [ 8.545585] Key type big_key registered [ 8.571284] Key type encrypted registered [ 8.573759] ima: No TPM chip found, activating TPM-bypass! [ 8.577782] ima: Allocated hash algorithm: sha1 [ 8.580449] ima: No architecture policies found [ 8.582821] evm: Initialising EVM extended attributes: [ 8.585672] evm: security.selinux [ 8.587316] evm: security.ima [ 8.589701] evm: security.capability [ 8.597311] evm: HMAC attrs: 0x1 [ 8.606272] rtc_cmos 00:05: setting system clock to 2026-09-08 21:30:02 UTC (1788903002) [ 8.624542] debug: unmapping init [mem 0xffffffffac403000-0xffffffffac5fffff] [ 8.638784] debug: unmapping init [mem 0xffffffffab182000-0xffffffffab458fff] [ 8.659201] Write protecting the kernel read-only data: 28672k [ 8.663548] debug: unmapping init [mem 0xffffffffa9803000-0xffffffffa99fffff] [ 8.674466] debug: unmapping init [mem 0xffffffffaa114000-0xffffffffaa1fffff] [ 8.761277] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 8.777355] systemd[1]: Detected virtualization kvm. [ 8.780280] systemd[1]: Detected architecture x86-64. [ 8.782714] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 8.826381] systemd[1]: No hostname configured. [ 8.832474] systemd[1]: Set hostname to . [ 8.837583] random: systemd: uninitialized urandom read (16 bytes read) [ 8.844274] systemd[1]: Initializing machine ID from random generator. [ 9.103615] random: systemd: uninitialized urandom read (16 bytes read) [ 9.109709] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 9.118577] random: systemd: uninitialized urandom read (16 bytes read) [ 9.122739] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 9.130134] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Journal Service... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 10.602262] device-mapper: uevent: version 1.0.3 [ 10.606739] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 12.353630] virtio_net virtio0 ens2: renamed from eth0 [ 12.446172] random: fast init done [ 12.750514] scsi host0: ata_piix [ 12.768797] scsi host1: ata_piix [ 12.771779] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 12.776751] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 17.113184] random: crng init done [ 17.116428] random: 7 urandom warning(s) missed due to ratelimiting [ 17.642484] dracut-initqueue[582]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 19.455771] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 22.046483] printk: systemd: 26 output lines suppressed due to ratelimiting [ 22.709943] SELinux: Disabled at runtime. [ 22.805149] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 22.817737] systemd[1]: Detected virtualization kvm. [ 22.820316] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 24.118459] systemd[1]: initrd-switch-root.service: Succeeded. [ 24.123413] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 24.136024] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 24.143199] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 24.147223] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 24.157387] systemd[1]: Starting Journal Service... Starting Journal Service... [ 24.170487] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 24.691865] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 25.621271] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 26.402302] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 26.535457] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 26.979228] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 27.034290] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit)[ 31.754909] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (7s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 32.583706] NFS: Registering the id_resolver key type [ 32.590532] Key type id_resolver registered [ 32.595222] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg338-server login: [ 54.403004] hrtimer: interrupt took 1839741 ns [ 87.165531] libcfs: loading out-of-tree module taints kernel. [ 87.232504] Key type ._llcrypt registered [ 87.235453] Key type .llcrypt registered [ 87.331172] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_hostid [ 105.379363] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 107.044294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 107.061568] alg: No test for adler32 (adler32-zlib) [ 108.552963] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 109.571866] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 111.359174] Key type lgssc registered [ 113.294926] Lustre: Echo OBD driver; http://www.lustre.org/ [ 131.585091] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 170.731807] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 183.820047] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 183.847414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 185.106535] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 185.143237] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 185.255612] Lustre: lustre-MDT0000: new disk, initializing [ 185.364015] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 185.398822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 189.858222] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 202.898789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 202.990245] Lustre: 6514:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 203.016593] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 203.022306] Lustre: Skipped 1 previous similar message [ 203.134396] Lustre: lustre-MDT0001: new disk, initializing [ 203.234294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 203.258951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 203.269604] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 207.902657] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.522366] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 220.901321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 221.225649] Lustre: lustre-OST0000: new disk, initializing [ 221.230617] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 221.237124] Lustre: 8452:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 221.335197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 226.852281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 226.865448] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 228.017955] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 231.928278] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 242.832438] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 242.974236] Lustre: lustre-OST0001: new disk, initializing [ 242.980814] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 242.989951] Lustre: 9525:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 243.072091] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 248.604670] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 249.422930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 249.434410] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 249.477897] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 259.740721] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 263.754868] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 268.177701] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing check_logdir /tmp/testlogs/ [ 272.766183] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing yml_node [ 276.970222] Lustre: DEBUG MARKER: Client: 2.17.55.26 [ 279.691374] Lustre: DEBUG MARKER: MDS: 2.17.55.26 [ 282.576641] Lustre: DEBUG MARKER: OSS: 2.17.55.26 [ 284.609505] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 8 17:34:36 EDT 2026 [ 309.712344] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 311.121752] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 331.232678] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 331.240723] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 331.255222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 331.744209] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 331.759668] Lustre: Skipped 1 previous similar message [ 331.770627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 331.773568] Lustre: Skipped 1 previous similar message [ 335.115135] Lustre: server umount lustre-MDT0000 complete [ 338.356093] LustreError: 10342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903332 with bad export cookie 4912457116965699536 [ 338.360898] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 338.367026] LustreError: 10342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 341.984398] LustreError: 6522:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 341.985752] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 341.988949] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 342.033747] LustreError: 6522:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 342.053483] Lustre: Skipped 2 previous similar messages [ 344.040034] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 344.040977] LustreError: 7745:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 344.060867] Lustre: Skipped 2 previous similar messages [ 344.096608] LustreError: 7745:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 344.811947] Lustre: server umount lustre-MDT0001 complete [ 349.327543] Lustre: server umount lustre-OST0000 complete [ 354.603185] Lustre: server umount lustre-OST0001 complete [ 363.049220] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_hostid [ 373.464460] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 412.238907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 412.391366] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 412.415364] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 412.486337] Lustre: lustre-MDT0000: new disk, initializing [ 412.568466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 412.622154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 416.414420] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 424.415930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 424.541411] Lustre: 15084:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 424.588376] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 424.591517] Lustre: Skipped 1 previous similar message [ 424.662354] Lustre: lustre-MDT0001: new disk, initializing [ 424.729100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 424.752183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 424.762393] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 428.145016] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 434.840858] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 438.633610] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 442.860876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 443.033239] Lustre: lustre-OST0000: new disk, initializing [ 443.036351] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 443.040543] Lustre: 17048:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 443.092777] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 444.508766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 444.515913] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 444.627683] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 446.816233] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 454.267069] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 457.862307] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 458.113432] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 461.537201] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 461.706958] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 465.378496] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 465.379993] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 465.384563] LustreError: Skipped 1 previous similar message [ 465.386275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 465.401148] Lustre: Skipped 1 previous similar message [ 465.414988] Lustre: Skipped 1 previous similar message [ 469.269516] Lustre: server umount lustre-OST0000 complete [ 475.618176] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 475.625428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 475.626819] Lustre: Skipped 1 previous similar message [ 475.637414] Lustre: Skipped 1 previous similar message [ 485.855128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 485.983138] Lustre: server umount lustre-MDT0000 complete [ 488.541908] LustreError: 15074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903482 with bad export cookie 4912457116965701069 [ 488.545168] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 488.548288] LustreError: 15074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 488.769632] Lustre: server umount lustre-MDT0001 complete [ 493.296226] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 17:38:05 (1788903485) [ 505.966736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 506.292720] LustreError: 19921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 506.340626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 509.378594] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 511.460436] LustreError: 19922:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 515.422981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 518.647365] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 523.196259] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 526.535356] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 527.760923] Lustre: Failing over lustre-MDT0000 [ 527.948733] Lustre: server umount lustre-MDT0000 complete [ 530.913824] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 530.916383] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 530.926217] LustreError: 19916:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 530.940395] LustreError: 19916:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 540.130022] LustreError: 19921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 540.141320] LustreError: 19921:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 543.678377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 543.750394] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 546.848564] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 549.346343] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 549.353094] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 549.365792] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 554.071252] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 558.563966] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 558.570090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 558.572595] Lustre: Skipped 2 previous similar messages [ 558.583577] Lustre: Skipped 4 previous similar messages [ 571.359133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 571.471269] Lustre: server umount lustre-MDT0000 complete [ 573.704142] LustreError: 19901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903567 with bad export cookie 4912457116965702483 [ 573.708140] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 573.715662] LustreError: 19901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 573.927483] Lustre: server umount lustre-MDT0001 complete [ 579.427832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 579.702409] LustreError: 23257:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 579.754766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 579.761679] Lustre: Skipped 2 previous similar messages [ 582.455413] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 588.363058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 591.514433] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 596.771910] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 600.697705] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 602.105517] Lustre: Failing over lustre-MDT0000 [ 602.239436] Lustre: server umount lustre-MDT0000 complete [ 604.130578] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 604.136088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 613.345875] LustreError: 23258:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 613.357981] LustreError: 23258:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 618.031264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 618.116150] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 618.299134] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 620.860910] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 623.586329] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 623.592212] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 623.598525] Lustre: Skipped 1 previous similar message [ 623.615357] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 627.230500] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 633.378818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 633.381593] Lustre: Skipped 7 previous similar messages [ 636.572058] Lustre: server umount lustre-MDT0000 complete [ 639.380245] LustreError: 23239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903633 with bad export cookie 4912457116965703421 [ 639.384601] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 639.393249] LustreError: 23239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 639.662494] Lustre: server umount lustre-MDT0001 complete [ 645.466852] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 17:40:37 (1788903637) [ 655.982355] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_hostid [ 662.629772] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 689.291257] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 695.763933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 695.902342] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 695.920238] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 695.970746] Lustre: lustre-MDT0000: new disk, initializing [ 696.025539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 698.126739] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 704.163497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 704.225695] Lustre: 29627:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 704.373618] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 706.486752] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 712.030709] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 715.178804] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 718.985984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 719.173769] Lustre: 31594:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 719.231057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 719.237780] Lustre: Skipped 4 previous similar messages [ 720.708175] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 720.736297] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 720.771500] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 720.787281] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 722.534132] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 729.040233] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 735.144100] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 751.590586] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 751.593207] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 751.596394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 751.604388] Lustre: Skipped 4 previous similar messages [ 751.611597] Lustre: Skipped 1 previous similar message [ 753.423625] Lustre: server umount lustre-OST0000 complete [ 762.251355] Lustre: server umount lustre-MDT0000 complete [ 764.669681] LustreError: 30430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903758 with bad export cookie 4912457116965703827 [ 764.680290] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 764.685551] LustreError: 30430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 764.991805] Lustre: server umount lustre-MDT0001 complete [ 773.165390] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 775.308225] Key type lgssc unregistered [ 775.535587] LNet: 33567:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 775.541159] LNetError: 33567:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 775.558461] LNet: Removed LNI 192.168.203.138@tcp [ 776.235144] Key type .llcrypt unregistered [ 776.239443] Key type ._llcrypt unregistered [ 793.244414] Key type ._llcrypt registered [ 793.246385] Key type .llcrypt registered [ 793.304676] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 793.997026] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 794.014945] alg: No test for adler32 (adler32-zlib) [ 794.951330] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 795.122340] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 796.751162] Key type lgssc registered [ 797.454894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 802.884243] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 802.903289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 804.263131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 807.379899] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 813.121733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 813.376780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 816.140726] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 821.474529] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 825.561133] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 830.141071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 830.341418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 834.187181] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 838.185932] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 838.576870] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 861.029138] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 17:44:13 (1788903853) [ 862.088610] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 863.075747] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 866.659822] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 17:44:19 (1788903859) [ 871.113084] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 17:44:23 (1788903863) [ 871.641648] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 875.990294] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 875.993415] Lustre: Skipped 2 previous similar messages [ 876.967352] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 17:44:29 (1788903869) [ 881.003393] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 881.006526] Lustre: Skipped 4 previous similar messages [ 888.484477] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 17:44:41 (1788903881) [ 889.805329] Lustre: *** cfs_fail_loc=131b, val=2*** [ 891.681144] Lustre: *** cfs_fail_loc=131b, val=2*** [ 895.977322] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 17:44:48 (1788903888) [ 897.024351] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 897.028170] Lustre: Skipped 9 previous similar messages [ 903.526900] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 17:44:56 (1788903896) [ 910.304414] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 910.308303] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 910.315985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 911.329464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 911.329669] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 911.339725] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 915.814574] Lustre: server umount lustre-OST0000 complete [ 921.057575] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 921.063058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 921.071456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 921.074426] Lustre: Skipped 1 previous similar message [ 926.690330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 926.696683] Lustre: Skipped 2 previous similar messages [ 931.809090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 931.814177] Lustre: Skipped 1 previous similar message [ 932.323438] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 932.413698] Lustre: server umount lustre-MDT0000 complete [ 934.144808] LustreError: 34940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903928 with bad export cookie 16224847749671414642 [ 934.150074] LustreError: 34940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 934.151884] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 934.321371] Lustre: server umount lustre-MDT0001 complete [ 939.132460] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 940.643400] Key type lgssc unregistered [ 940.827780] LNet: 42815:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 940.833032] LNetError: 42815:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 940.845295] LNet: Removed LNI 192.168.203.138@tcp [ 941.337056] Key type .llcrypt unregistered [ 941.339227] Key type ._llcrypt unregistered [ 950.637844] Key type ._llcrypt registered [ 950.640304] Key type .llcrypt registered [ 950.698251] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_hostid [ 959.587590] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 960.030886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 960.123886] alg: No test for adler32 (adler32-zlib) [ 961.063114] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 961.196143] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 962.807169] Key type lgssc registered [ 963.432670] Lustre: Echo OBD driver; http://www.lustre.org/ [ 980.838150] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 980.851537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 981.979740] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 981.999854] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 982.048171] Lustre: lustre-MDT0000: new disk, initializing [ 982.094452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 982.108220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 984.139207] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 989.761484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 989.827366] Lustre: 45942:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 989.867211] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 989.870560] Lustre: Skipped 1 previous similar message [ 989.911835] Lustre: lustre-MDT0001: new disk, initializing [ 989.974271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 989.999720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 990.007081] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 992.084069] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 997.222669] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 999.637076] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1002.606218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1002.729476] Lustre: lustre-OST0000: new disk, initializing [ 1002.731610] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1002.735461] Lustre: 47908:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1002.777391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1005.410442] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1010.392648] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1012.628755] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1012.743081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1012.750142] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1012.807424] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1013.811703] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 1016.569896] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1016.698442] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1022.947793] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1022.947914] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1022.948863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1022.955990] Lustre: Skipped 1 previous similar message [ 1028.067781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1028.071359] Lustre: Skipped 2 previous similar messages [ 1032.159130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1032.240989] Lustre: server umount lustre-OST0000 complete [ 1036.257452] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1036.263980] LustreError: Skipped 1 previous similar message [ 1036.267110] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1036.281314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1038.306601] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1038.310508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1038.316066] Lustre: Skipped 1 previous similar message [ 1040.267414] Lustre: server umount lustre-MDT0000 complete [ 1042.185179] LustreError: 45933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904036 with bad export cookie 2031622879836627598 [ 1042.186432] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1042.192664] LustreError: 45933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1042.365398] Lustre: server umount lustre-MDT0001 complete [ 1047.361209] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 17:47:20 (1788904040) [ 1050.989120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1051.259715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1053.517289] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1056.738314] LustreError: 50376:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1056.749188] LustreError: 50376:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1058.568737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1058.791739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1061.451372] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1065.447408] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1068.422541] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1071.894228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1072.065065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1075.039632] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1078.732934] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1085.624883] LustreError: 52903:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1086.151567] Lustre: Modifying parameter general.timeout=1 in log params [ 1087.555689] Lustre: Modifying parameter general.timeout=4 in log params [ 1087.561310] Lustre: Skipped 2 previous similar messages [ 1089.980401] Lustre: Modifying parameter general.timeout=9 in log params [ 1089.986091] Lustre: Skipped 4 previous similar messages [ 1094.490372] Lustre: Modifying parameter general.timeout=18 in log params [ 1094.494018] Lustre: Skipped 8 previous similar messages [ 1102.800907] Lustre: Modifying parameter general.timeout=34 in log params [ 1102.803884] Lustre: Skipped 15 previous similar messages [ 1118.926367] Lustre: Modifying parameter general.timeout=70 in log params [ 1118.929649] Lustre: Skipped 35 previous similar messages [ 1151.918125] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 17:49:04 (1788904144) [ 1152.375525] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_policies=tbf in log params [ 1152.379363] Lustre: Skipped 60 previous similar messages [ 1153.226763] Lustre: 59811:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1155.051312] Lustre: 60008:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1155.061857] Lustre: 60008:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1157.097874] Lustre: 60221:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1157.104525] Lustre: 60221:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1181.066655] Lustre: 61676:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1181.071532] Lustre: 61676:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1181.088893] LustreError: 61676:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1184.676627] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 17:49:36 (1788904176) [ 1196.007050] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1196.014326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1196.025505] Lustre: Skipped 1 previous similar message [ 1196.038945] Lustre: Skipped 2 previous similar messages [ 1201.841613] Lustre: server umount lustre-MDT0000 complete [ 1204.685464] LustreError: 50357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904198 with bad export cookie 2031622879836628466 [ 1204.690971] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1204.692301] LustreError: 50357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1204.942742] Lustre: server umount lustre-MDT0001 complete [ 1217.304597] Lustre: server umount lustre-OST0000 complete [ 1239.544801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1239.997250] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1240.087580] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1240.092649] Lustre: Skipped 21 previous similar messages [ 1241.403319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1246.507721] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1258.213775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1258.423471] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1258.489744] Lustre: 64203:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1265.329253] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1274.034643] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1281.736952] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1289.659504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1290.012368] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1290.529995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1290.533135] Lustre: Skipped 1 previous similar message [ 1298.457776] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1305.254748] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1306.749204] Lustre: 66772:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1316.324694] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1316.330602] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1316.342638] Lustre: Skipped 1 previous similar message [ 1316.349557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1316.356933] Lustre: Skipped 3 previous similar messages [ 1320.368703] Lustre: server umount lustre-OST0000 complete [ 1325.540908] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1325.557631] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1325.571356] Lustre: Skipped 1 previous similar message [ 1336.289443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1336.296204] Lustre: Skipped 7 previous similar messages [ 1338.335122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1338.544277] Lustre: server umount lustre-MDT0000 complete [ 1341.412294] LustreError: 64211:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1341.426263] LustreError: 64211:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1341.713746] LustreError: 64196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904335 with bad export cookie 2031622879836631028 [ 1341.717624] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1341.728954] LustreError: 64196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1342.143188] Lustre: server umount lustre-MDT0001 complete [ 1352.759760] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 1355.596490] Key type lgssc unregistered [ 1355.917651] LNet: 67982:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1355.932301] LNetError: 67982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1355.945358] LNet: Removed LNI 192.168.203.138@tcp [ 1356.871953] Key type .llcrypt unregistered [ 1356.875086] Key type ._llcrypt unregistered [ 1378.975587] Key type ._llcrypt registered [ 1378.978503] Key type .llcrypt registered [ 1379.067823] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 1380.153324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1380.197782] alg: No test for adler32 (adler32-zlib) [ 1381.167625] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1381.433104] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 1383.159759] Key type lgssc registered [ 1384.250724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1391.332113] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1391.358463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1392.808450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1396.202321] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1402.340029] LustreError: 69397:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1402.359101] LustreError: 69397:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1403.281226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1403.852984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1407.992994] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1414.689369] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1419.273910] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1425.181735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1425.641585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1431.285691] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1438.134627] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1446.324091] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1453.536402] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1453.543631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1453.560612] Lustre: Skipped 1 previous similar message [ 1453.572283] Lustre: Skipped 2 previous similar messages [ 1458.658148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1458.663426] Lustre: Skipped 2 previous similar messages [ 1459.814733] Lustre: server umount lustre-MDT0000 complete [ 1463.068165] LustreError: 69379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904456 with bad export cookie 8714304775608742164 [ 1463.078150] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1463.367876] Lustre: server umount lustre-MDT0001 complete [ 1477.270187] Lustre: server umount lustre-OST0000 complete [ 1502.336455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1502.594804] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1502.615815] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1502.823189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1507.173559] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1515.171611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1515.252703] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1515.288210] Lustre: 74265:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1515.377117] LustreError: 74276:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1515.751266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1519.727680] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1528.034965] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1533.689168] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1541.086669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1541.228663] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1541.603468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1548.340847] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1555.864855] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1557.837593] Lustre: 76839:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1559.688642] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1566.694996] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1566.703906] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1566.727152] Lustre: Skipped 1 previous similar message [ 1566.732160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1567.209559] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1571.002284] Lustre: server umount lustre-OST0000 complete [ 1576.932798] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1576.945917] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1576.971508] Lustre: Skipped 1 previous similar message [ 1576.981924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1576.990640] Lustre: Skipped 2 previous similar messages [ 1581.763782] Lustre: server umount lustre-MDT0000 complete [ 1582.568219] LustreError: 74276:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1582.588186] LustreError: 74276:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1585.962755] LustreError: 74258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904579 with bad export cookie 8714304775608743004 [ 1585.973711] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1585.979649] LustreError: 74258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 1586.479244] Lustre: server umount lustre-MDT0001 complete [ 1597.064882] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 1600.063348] Key type lgssc unregistered [ 1600.450797] LNet: 78052:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1600.460505] LNetError: 78052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1600.484982] LNet: Removed LNI 192.168.203.138@tcp [ 1601.561098] Key type .llcrypt unregistered [ 1601.565424] Key type ._llcrypt unregistered [ 1630.073489] Key type ._llcrypt registered [ 1630.079410] Key type .llcrypt registered [ 1630.223224] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 1631.400273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1631.438626] alg: No test for adler32 (adler32-zlib) [ 1632.630233] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1632.948886] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 1634.624233] Key type lgssc registered [ 1635.950346] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1644.454466] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1644.475178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1646.109198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1651.614804] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1655.269326] LustreError: 79470:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1655.301572] LustreError: 79470:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1660.387574] LustreError: 79471:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1663.756973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1665.150897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1671.514235] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1683.816384] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1689.423341] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1696.162058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1696.817096] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1704.376149] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1714.017594] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1725.294209] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1726.369895] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1727.526711] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1730.571944] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1730.581260] Lustre: Skipped 2 previous similar messages [ 1735.433844] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1735.437692] Lustre: Skipped 4 previous similar messages [ 1744.161686] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1744.169963] Lustre: Skipped 9 previous similar messages [ 1760.517357] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1760.526679] Lustre: Skipped 20 previous similar messages [ 1792.707534] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1792.712518] Lustre: Skipped 43 previous similar messages [ 1809.829554] Lustre: 87568:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1811.844216] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 18:00:04 (1788904804) [ 1812.934564] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1814.198317] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 18:00:06 (1788904806) [ 1817.572244] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1817.573797] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1817.585163] LustreError: Skipped 1 previous similar message [ 1817.586648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1817.602827] Lustre: Skipped 1 previous similar message [ 1817.622809] Lustre: Skipped 1 previous similar message [ 1822.693571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1822.704447] Lustre: Skipped 1 previous similar message [ 1827.814814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1827.823699] Lustre: Skipped 1 previous similar message [ 1830.879123] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1831.051988] Lustre: server umount lustre-OST0000 complete [ 1838.049440] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1838.051967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1838.064951] Lustre: Skipped 1 previous similar message [ 1840.317518] Lustre: server umount lustre-MDT0000 complete [ 1842.942931] LustreError: 79450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904836 with bad export cookie 11105354793221388395 [ 1842.943954] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1842.952401] LustreError: 79450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1843.157906] Lustre: server umount lustre-MDT0001 complete [ 1851.226877] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 1853.539596] Key type lgssc unregistered [ 1853.826526] LNet: 88815:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1853.845023] LNetError: 88815:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1853.864824] LNet: Removed LNI 192.168.203.138@tcp [ 1854.704518] Key type .llcrypt unregistered [ 1854.706330] Key type ._llcrypt unregistered [ 1858.541816] Key type ._llcrypt registered [ 1858.543637] Key type .llcrypt registered [ 1858.612994] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1862.254944] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules [ 1863.175384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1863.194073] alg: No test for adler32 (adler32-zlib) [ 1864.186986] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1864.376548] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 1866.039553] Key type lgssc registered [ 1867.042672] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1868.243956] LustreError: 90030:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 1882.487540] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 1889.031844] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1889.876515] LustreError: 90030:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1889.920869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1891.307852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1894.097775] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1900.244373] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 18:01:32 (1788904892) [ 1901.539391] LustreError: 91052:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1901.563689] LustreError: 91052:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1903.512427] LustreError: 91085:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1903.521655] LustreError: 91085:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1903.528055] LustreError: 91085:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 12, retries 0, failed: rc = -5 [ 1903.754857] Lustre: server umount lustre-MDT0000 complete [ 1912.599700] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 1914.606277] Key type lgssc unregistered [ 1914.834657] LNet: 92578:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1914.841546] LNetError: 92578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1914.855460] LNet: Removed LNI 192.168.203.138@tcp [ 1915.390430] Key type .llcrypt unregistered [ 1915.392111] Key type ._llcrypt unregistered [ 1928.311689] Key type ._llcrypt registered [ 1928.314091] Key type .llcrypt registered [ 1928.417578] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 1929.156482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1929.243123] alg: No test for adler32 (adler32-zlib) [ 1930.287689] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1930.481821] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 1932.111184] Key type lgssc registered [ 1932.913680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1938.322669] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1938.336871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1939.710695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1942.748309] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1948.209150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1948.690072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1951.763494] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1957.062741] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1961.473576] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1966.021851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1966.488285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1970.479255] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 1975.853172] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2019.676805] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 18:03:32 (1788905012) [ 2020.708512] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2021.805856] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 18:03:34 (1788905014) [ 2025.441052] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2025.446643] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2025.462696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2025.955589] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2025.969874] Lustre: Skipped 1 previous similar message [ 2025.973341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2025.977208] Lustre: Skipped 1 previous similar message [ 2030.286962] Lustre: server umount lustre-MDT0000 complete [ 2031.076053] LustreError: 94220:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2031.096977] LustreError: 94220:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2032.615705] LustreError: 97035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905026 with bad export cookie 12072836481644436791 [ 2032.620683] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2032.626671] LustreError: 97035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2032.798473] Lustre: server umount lustre-MDT0001 complete [ 2044.427976] Lustre: server umount lustre-OST0000 complete [ 2051.097198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2051.338055] LustreError: 98066:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2051.465980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2053.901684] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2056.674350] LustreError: 98067:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2059.029927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2059.422930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2062.029719] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2066.702871] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2070.295601] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2078.340985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2078.488888] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2078.494194] LustreError: 98054:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2078.504917] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2078.508512] LustreError: 100146:0:(tgt_mount.c:2703:server_fill_super()) Unable to start targets: -98 [ 2078.513208] LustreError: 100146:0:(tgt_mount.c:2221:server_put_super()) no obd lustre-OST0000 [ 2078.518127] LustreError: 100146:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2078.557427] Lustre: server umount lustre-OST0000 complete [ 2078.559798] LustreError: 100146:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2083.998639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2084.106712] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2084.111825] LustreError: 98054:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2084.117967] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2084.123389] LustreError: 100608:0:(tgt_mount.c:2703:server_fill_super()) Unable to start targets: -98 [ 2084.127605] LustreError: 100608:0:(tgt_mount.c:2221:server_put_super()) no obd lustre-OST0000 [ 2084.131539] LustreError: 100608:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2084.169805] Lustre: server umount lustre-OST0000 complete [ 2084.172308] LustreError: 100608:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2089.803422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2089.914239] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2089.918548] Lustre: Found index 0 for lustre-OST0000, updating log [ 2089.923953] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2089.977736] Lustre: lustre-OST0000: new disk, initializing [ 2089.982288] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2089.987710] Lustre: 101142:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2090.151249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2093.888348] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2096.158119] Lustre: Failing over lustre-OST0000 [ 2096.248505] Lustre: server umount lustre-OST0000 complete [ 2100.192088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2100.197912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2100.206770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2102.242863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2102.243111] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2104.820061] Lustre: server umount lustre-MDT0000 complete [ 2106.936723] LustreError: 98047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905100 with bad export cookie 12072836481644439535 [ 2106.942923] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2106.949591] LustreError: 98047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2107.173126] Lustre: server umount lustre-MDT0001 complete [ 2112.016577] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 18:05:04 (1788905104) [ 2121.670983] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 2127.421568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2127.628034] LustreError: 103483:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2127.693096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2130.039266] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2132.964667] LustreError: 103484:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2134.526798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2137.037445] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2138.611414] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2141.978196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2145.530249] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2149.975686] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2150.030281] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2150.088546] Lustre: lustre-OST0001: new disk, initializing [ 2150.092706] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2150.097691] Lustre: 105799:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2153.417955] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2156.564091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2156.570056] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2156.621832] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2157.735389] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2163.331564] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2166.753121] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2166.759933] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2166.766516] Lustre: Skipped 2 previous similar messages [ 2166.772177] Lustre: Skipped 5 previous similar messages [ 2171.578682] Lustre: server umount lustre-MDT0001 complete [ 2171.877134] LustreError: 104214:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2171.890550] LustreError: 104214:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2176.403607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2176.479172] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2176.482161] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2176.506355] Lustre: 103472:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2176.887090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2176.891153] Lustre: Skipped 3 previous similar messages [ 2177.767786] LustreError: 107404:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2177.772624] LustreError: 107404:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2177.776790] LustreError: 103475:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2177.816748] LustreError: 107421:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.138@tcp: cfg command failed: rc = -17 [ 2177.822466] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2177.822466] [ 2178.665394] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2182.119354] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2182.659833] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 18:06:15 (1788905175) [ 2189.892454] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 2197.394779] Lustre: 109410:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2207.545459] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2209.069298] Lustre: 110543:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2264.031720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2264.032140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2264.038283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2264.044481] Lustre: Skipped 4 previous similar messages [ 2269.081355] Lustre: server umount lustre-MDT0000 complete [ 2269.152225] LustreError: 104556:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2269.160065] LustreError: 104556:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2270.645150] LustreError: 103464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905264 with bad export cookie 12072836481644440445 [ 2270.646184] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2270.651199] LustreError: 103464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2270.788408] Lustre: server umount lustre-MDT0001 complete [ 2282.506911] Lustre: server umount lustre-OST0000 complete [ 2294.114847] Lustre: server umount lustre-OST0001 complete [ 2296.203218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2299.589563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2302.502661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2305.670136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2312.064512] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 2315.604805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2315.611478] Lustre: lustre-MDT0000: reset Object Index mappings [ 2315.755735] LustreError: 114487:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2315.784136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2317.187842] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2320.152919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2320.159291] Lustre: lustre-MDT0001: reset Object Index mappings [ 2321.827300] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2322.845412] Lustre: 115634:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2325.028499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2326.310941] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 2327.185270] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2330.087339] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2331.371988] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:157 to 0x2c0000401:193) [ 2331.373365] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 2331.407628] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:199 to 0x2c0000400:225) [ 2332.284952] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2335.461988] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2336.557982] Lustre: 117509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2350.477969] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 18:09:03 (1788905343) [ 2381.791885] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2381.793912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2381.798735] Lustre: Skipped 3 previous similar messages [ 2381.801139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2385.948287] Lustre: server umount lustre-MDT0000 complete [ 2387.032065] LustreError: 114521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905380 with bad export cookie 12072836481644487240 [ 2387.035698] LustreError: 114521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2387.037187] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2387.936057] LustreError: 114483:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2387.940803] LustreError: 114483:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2393.406343] Lustre: server umount lustre-MDT0001 complete [ 2400.899506] Lustre: server umount lustre-OST0000 complete [ 2408.219198] Lustre: server umount lustre-OST0001 complete [ 2410.225947] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_hostid [ 2413.388588] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 2424.825243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2424.900918] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2424.911643] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2424.947563] Lustre: lustre-MDT0000: new disk, initializing [ 2424.973690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2426.269313] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2429.078190] Lustre: server umount lustre-MDT0000 complete [ 2432.078922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2432.179411] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2432.186050] Lustre: 121762:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2432.192309] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2433.558832] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2435.186379] Lustre: server umount lustre-MDT0000 complete [ 2437.883213] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 18:10:30 (1788905430) [ 2438.398234] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2438.992914] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 18:10:32 (1788905432) [ 2444.431488] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 2447.689929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2447.806870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2447.810192] Lustre: Skipped 5 previous similar messages [ 2449.110733] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2451.715573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2451.753819] Lustre: 123813:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2451.779403] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2451.782389] Lustre: Skipped 1 previous similar message [ 2451.822435] Lustre: lustre-MDT0001: new disk, initializing [ 2451.867710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2451.872325] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2453.218332] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2455.487603] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2457.551015] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2457.642600] Lustre: lustre-OST0000: new disk, initializing [ 2457.644246] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2457.646491] Lustre: 125386:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2458.055500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2458.058546] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2458.085654] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2459.441217] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2463.452083] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2463.514370] Lustre: 126257:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2464.773426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2464.777240] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2464.806964] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2465.388326] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2469.645419] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2470.794306] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2587.937443] Lustre: 132627:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 2587.942458] Lustre: 132627:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 2587.947391] Lustre: 132627:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 2587.952578] Lustre: 132627:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 2587.957772] Lustre: 132627:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 2587.963669] Lustre: 132627:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2751.221987] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 18:15:43 (1788905743) [ 2764.257440] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2764.261891] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2764.268852] Lustre: Skipped 5 previous similar messages [ 2764.272217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2764.276199] Lustre: Skipped 8 previous similar messages [ 2769.604090] Lustre: server umount lustre-MDT0000 complete [ 2771.246213] LustreError: 123844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905765 with bad export cookie 12072836481644539768 [ 2771.252525] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2771.253063] LustreError: 123844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2771.936494] LustreError: 127057:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2771.946690] LustreError: 127057:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2777.543852] Lustre: server umount lustre-MDT0001 complete [ 2785.539259] Lustre: server umount lustre-OST0000 complete [ 2793.390424] Lustre: server umount lustre-OST0001 complete [ 2797.443364] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_hostid [ 2803.569865] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 2825.156316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2825.405344] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2825.474640] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2825.478536] Lustre: Skipped 1 previous similar message [ 2825.536779] Lustre: lustre-MDT0000: new disk, initializing [ 2825.539061] Lustre: Skipped 1 previous similar message [ 2825.614454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2825.618514] Lustre: Skipped 3 previous similar messages [ 2825.640533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2828.707181] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2835.699183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2835.780834] Lustre: 135656:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2835.996840] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2838.682365] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2844.974590] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2848.674203] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2852.964486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2853.175703] Lustre: lustre-OST0000: new disk, initializing [ 2853.178270] Lustre: Skipped 1 previous similar message [ 2853.181127] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2853.184624] Lustre: Skipped 2 previous similar messages [ 2853.188901] Lustre: 137625:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2854.809485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2854.814916] Lustre: Skipped 1 previous similar message [ 2854.820907] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2854.899475] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2856.806773] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 2862.724686] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2868.056387] Lustre: lustre-MDD0000: changelog on [ 2869.619885] Lustre: lustre-MDD0001: changelog on [ 4199.857962] Lustre: lustre-MDD0001: changelog off [ 4200.706107] Lustre: 135662:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 4201.099525] Lustre: lustre-MDD0000: changelog off [ 4206.560172] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4206.563657] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4206.564662] LustreError: Skipped 1 previous similar message [ 4206.574992] Lustre: Skipped 5 previous similar messages [ 4206.577461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4206.583022] Lustre: Skipped 9 previous similar messages [ 4209.449272] Lustre: server umount lustre-OST0000 complete [ 4216.801248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4216.805186] Lustre: Skipped 3 previous similar messages [ 4217.489519] Lustre: server umount lustre-MDT0000 complete [ 4219.177638] LustreError: 137624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907213 with bad export cookie 12072836481645336970 [ 4219.179721] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4219.186418] LustreError: 137624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4219.354722] Lustre: server umount lustre-MDT0001 complete [ 4224.010414] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 4225.253653] Key type lgssc unregistered [ 4225.411157] LNet: 146691:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4225.416435] LNetError: 146691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4225.431277] LNet: Removed LNI 192.168.203.138@tcp [ 4225.825190] Key type .llcrypt unregistered [ 4225.830558] Key type ._llcrypt unregistered [ 4234.222269] Key type ._llcrypt registered [ 4234.224723] Key type .llcrypt registered [ 4234.273575] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_hostid [ 4243.191697] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 4243.659506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4243.729516] alg: No test for adler32 (adler32-zlib) [ 4244.650901] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 4244.790208] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 4246.415173] Key type lgssc registered [ 4246.960634] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4264.057869] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4264.075442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4265.215023] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4265.234479] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4265.289630] Lustre: lustre-MDT0000: new disk, initializing [ 4265.344614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4265.357837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4267.424279] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4273.030823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4273.101123] Lustre: 149865:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4273.153389] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4273.157908] Lustre: Skipped 1 previous similar message [ 4273.202425] Lustre: lustre-MDT0001: new disk, initializing [ 4273.254184] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4273.272411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4273.278442] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4275.494569] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4281.669490] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4286.350744] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4296.166355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4296.631509] Lustre: lustre-OST0000: new disk, initializing [ 4296.636678] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4296.642984] Lustre: 151835:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4296.813939] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4303.744410] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4306.542524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4306.557229] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4306.731930] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4315.950850] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4324.423154] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_module obdecho/obdecho [ 4326.968259] LustreError: 152839:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 4327.005162] LustreError: 152839:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 4328.937905] LustreError: 152935:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 4342.755049] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4342.759415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4342.772995] Lustre: Skipped 2 previous similar messages [ 4342.795146] Lustre: Skipped 2 previous similar messages [ 4347.749877] Lustre: server umount lustre-MDT0000 complete [ 4353.006430] LustreError: 149874:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4353.022283] LustreError: 149858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907346 with bad export cookie 732794108296213839 [ 4353.038912] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4353.087189] LustreError: 149874:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4353.092351] LustreError: 149858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4353.526495] Lustre: server umount lustre-MDT0001 complete [ 4369.217644] Lustre: server umount lustre-OST0000 complete [ 4388.099502] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_hostid [ 4396.223265] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 4421.167422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4421.392147] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4421.415277] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4421.468733] Lustre: lustre-MDT0000: new disk, initializing [ 4421.521667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4421.542797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4424.912273] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4432.796662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4432.880312] Lustre: 156646:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4432.948210] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4432.954039] Lustre: Skipped 1 previous similar message [ 4433.037265] Lustre: lustre-MDT0001: new disk, initializing [ 4433.158620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4433.201707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4433.216092] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4436.067935] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4443.254491] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4447.006359] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4451.321852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4451.486365] Lustre: lustre-OST0000: new disk, initializing [ 4451.492955] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4451.499517] Lustre: 158616:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4451.569061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4455.569810] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4458.017499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4458.029587] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4458.101166] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4461.606735] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4464.399284] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4464.527843] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4467.820772] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4468.010688] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4473.316916] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4473.321147] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4473.344034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4473.349334] LustreError: Skipped 1 previous similar message [ 4473.361592] Lustre: Skipped 1 previous similar message [ 4475.701638] Lustre: server umount lustre-OST0000 complete [ 4479.456264] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4479.466961] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4479.481270] Lustre: Skipped 1 previous similar message [ 4479.488240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4483.564435] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4483.566683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4483.590290] Lustre: Skipped 1 previous similar message [ 4488.673535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4488.681087] Lustre: Skipped 1 previous similar message [ 4493.279140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4493.497289] Lustre: server umount lustre-MDT0000 complete [ 4493.794527] LustreError: 157807:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4493.813421] LustreError: 157807:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4496.846337] LustreError: 156640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907490 with bad export cookie 732794108296214798 [ 4496.851581] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4496.852762] LustreError: 156640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4497.098648] Lustre: server umount lustre-MDT0001 complete [ 4500.065324] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 18:44:52 (1788907492) [ 4501.371449] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 4502.881295] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 18:44:55 (1788907495) [ 4509.366397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4509.696241] LustreError: 161026:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4509.757354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4513.637965] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4514.785848] LustreError: 161027:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4519.907761] LustreError: 161026:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4520.043901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4524.051240] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4530.250647] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4534.989143] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4540.114403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4545.594322] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4552.071623] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4564.961794] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4564.967196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4564.973604] Lustre: Skipped 2 previous similar messages [ 4564.981887] Lustre: Skipped 2 previous similar messages [ 4569.228840] Lustre: server umount lustre-MDT0000 complete [ 4570.085765] LustreError: 161027:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4570.105767] LustreError: 161027:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4572.604629] LustreError: 161008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907566 with bad export cookie 732794108296215680 [ 4572.626792] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4572.627737] LustreError: 161008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4575.202868] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4578.910704] Lustre: server umount lustre-MDT0001 complete [ 4583.740040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4591.680398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4591.829823] LustreError: 164866:0:(llog.c:1655:llog_backup()) MGC192.168.203.138@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 4591.837230] Lustre: 164866:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.138@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 4602.847991] LustreError: 164871:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4602.861748] LustreError: 164871:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4637.665623] LustreError: 164892:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4637.682246] LustreError: 164892:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 29 previous similar messages [ 4653.023468] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 4653.029924] Lustre: 164866:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.138@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 4693.549967] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4693.553697] Lustre: Skipped 2 previous similar messages [ 4695.171337] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4697.570603] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4698.366753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4702.688751] LustreError: 164904:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4702.695706] LustreError: 164904:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 42 previous similar messages [ 4719.773418] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4722.703700] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4723.173946] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4723.395080] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4725.762964] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4726.420321] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4733.410880] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4733.412308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4733.417388] Lustre: Skipped 2 previous similar messages [ 4733.419912] Lustre: Skipped 6 previous similar messages [ 4733.424334] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4738.728537] Lustre: server umount lustre-MDT0000 complete [ 4740.098590] LustreError: 161007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907733 with bad export cookie 732794108296222169 [ 4740.101726] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4740.104290] LustreError: 161007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4740.227109] Lustre: server umount lustre-MDT0001 complete [ 4751.902163] Lustre: server umount lustre-OST0000 complete [ 4759.479142] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_hostid [ 4763.092179] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 4778.549868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4778.667553] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4778.680797] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4778.722139] Lustre: lustre-MDT0000: new disk, initializing [ 4778.758228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4778.760978] Lustre: Skipped 1 previous similar message [ 4778.770775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4780.641946] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4785.597595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4785.645578] Lustre: 169943:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4785.772755] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4787.461771] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4791.863779] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4794.166113] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4796.909545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4797.024972] Lustre: 171909:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4799.426973] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4803.744305] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4804.610459] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4804.656778] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4805.503666] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4805.604608] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4807.427436] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4807.518206] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4809.700919] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4809.701273] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4809.708277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4809.709985] LustreError: Skipped 1 previous similar message [ 4809.713617] Lustre: Skipped 5 previous similar messages [ 4810.430060] Lustre: server umount lustre-OST0000 complete [ 4818.346079] Lustre: server umount lustre-MDT0000 complete [ 4819.980229] LustreError: 169935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907813 with bad export cookie 732794108296238976 [ 4819.980831] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4819.986512] LustreError: 169935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4820.115422] Lustre: server umount lustre-MDT0001 complete [ 4823.742401] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 18:50:16 (1788907816) [ 4826.496209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4828.324896] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4831.619216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4833.474668] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4836.418964] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4838.654211] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4841.452616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4844.001667] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4847.398510] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4857.716267] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 18:50:50 (1788907850) [ 4863.458633] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4863.461792] LustreError: Skipped 1 previous similar message [ 4865.381057] Lustre: server umount lustre-OST0000 complete [ 4878.817173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4878.820111] Lustre: Skipped 10 previous similar messages [ 4881.375115] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4881.432642] Lustre: server umount lustre-MDT0000 complete [ 4882.790552] LustreError: 174297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907876 with bad export cookie 732794108296239837 [ 4882.795835] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4882.796625] LustreError: 174297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4882.920291] Lustre: server umount lustre-MDT0001 complete [ 4887.344996] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 4888.600817] Key type lgssc unregistered [ 4888.756539] LNet: 178038:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4888.759810] LNetError: 178038:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4888.769620] LNet: Removed LNI 192.168.203.138@tcp [ 4889.170645] Key type .llcrypt unregistered [ 4889.172416] Key type ._llcrypt unregistered [ 4898.048520] Key type ._llcrypt registered [ 4898.050336] Key type .llcrypt registered [ 4898.103502] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 4898.511860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4898.572041] alg: No test for adler32 (adler32-zlib) [ 4899.462604] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 4899.570272] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 4901.159219] Key type lgssc registered [ 4901.663114] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4904.710979] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4904.719224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4920.159413] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4925.279483] LustreError: 179258:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.138@tcp: failed processing log, type 4: rc = -110 [ 4930.401247] LustreError: 179258:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4930.407321] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 4930.410458] LustreError: MGC192.168.203.138@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 4930.417036] LustreError: 179258:0:(tgt_mount.c:1966:server_start_targets()) failed to start server lustre-OST0000: -22 [ 4930.422082] LustreError: 179258:0:(tgt_mount.c:2703:server_fill_super()) Unable to start targets: -22 [ 4930.425631] LustreError: 179258:0:(tgt_mount.c:2221:server_put_super()) no obd lustre-OST0000 [ 4930.448850] Lustre: server umount lustre-OST0000 complete [ 4930.451234] LustreError: 179258:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4934.029597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4935.223601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4936.874982] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4940.295872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4940.440795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4942.170219] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4944.978931] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4947.226731] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4949.809044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4949.896600] LustreError: 181570:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4949.901578] Lustre: 181570:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.138@tcp: invalid local config log lustre-OST0000: rc = -22 [ 4949.959715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4952.144498] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 4955.237194] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4956.130049] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4956.130616] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4956.131992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4956.136242] Lustre: Skipped 1 previous similar message [ 4961.250362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4961.253733] Lustre: Skipped 4 previous similar messages [ 4962.208714] Lustre: server umount lustre-MDT0000 complete [ 4963.707356] LustreError: 182332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907957 with bad export cookie 5797465035491170926 [ 4963.710826] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4963.712955] LustreError: 182332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4963.851886] Lustre: server umount lustre-MDT0001 complete [ 4975.726095] Lustre: server umount lustre-OST0000 complete [ 4979.548982] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_hostid [ 4983.994716] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 4998.026412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4998.135837] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4998.149256] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4998.188526] Lustre: lustre-MDT0000: new disk, initializing [ 4998.227376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4998.244287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4999.758158] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5004.477712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5004.529840] Lustre: 185091:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5004.566257] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5004.569371] Lustre: Skipped 1 previous similar message [ 5004.609924] Lustre: lustre-MDT0001: new disk, initializing [ 5004.652327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5004.667077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5004.671392] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5006.275633] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5010.440405] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5012.515478] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5014.972664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5015.083602] Lustre: lustre-OST0000: new disk, initializing [ 5015.086393] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5015.090554] Lustre: 187059:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5015.123111] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5017.417375] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5020.665628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5020.670434] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5020.708882] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5021.490612] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5023.010631] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5023.084353] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5024.451905] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5024.519271] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5025.761511] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5025.762529] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5025.764698] LustreError: Skipped 1 previous similar message [ 5025.769940] Lustre: Skipped 1 previous similar message [ 5025.772516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5025.775488] Lustre: Skipped 1 previous similar message [ 5027.218099] Lustre: server umount lustre-OST0000 complete [ 5030.367923] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5030.371723] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5030.377161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5034.654632] Lustre: server umount lustre-MDT0000 complete [ 5036.000023] LustreError: 186461:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5036.007198] LustreError: 186461:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5036.082183] LustreError: 185887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908029 with bad export cookie 5797465035491171612 [ 5036.085487] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5036.086850] LustreError: 185887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5036.217559] Lustre: server umount lustre-MDT0001 complete [ 5039.395296] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 18:53:52 (1788908032) [ 5045.911439] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 5047.005977] Key type lgssc unregistered [ 5047.157342] LNet: 189998:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5047.160396] LNetError: 189998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5047.167407] LNet: Removed LNI 192.168.203.138@tcp [ 5047.487957] Key type .llcrypt unregistered [ 5047.489842] Key type ._llcrypt unregistered [ 5055.165478] Key type ._llcrypt registered [ 5055.167329] Key type .llcrypt registered [ 5055.216044] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 5055.563018] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5055.631545] alg: No test for adler32 (adler32-zlib) [ 5056.499483] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 5056.597625] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 5058.191165] Key type lgssc registered [ 5058.598330] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5061.531079] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5061.537601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5062.705365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5064.252526] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5067.522762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5067.694464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5069.345116] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5072.300121] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5074.444283] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5077.450565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5077.557840] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5077.559374] LustreError: 193054:0:(llog.c:1696:llog_backup()) MGC192.168.203.138@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5077.563964] Lustre: 193054:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.138@tcp: can't backup local config lustre-OST0000: rc = -28 [ 5077.617983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5079.901385] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5082.801136] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5083.569788] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5083.570361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5083.575808] Lustre: Skipped 1 previous similar message [ 5088.224084] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5088.227069] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5088.231785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5088.234075] Lustre: Skipped 1 previous similar message [ 5089.687569] Lustre: server umount lustre-MDT0000 complete [ 5090.991151] LustreError: 191173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908084 with bad export cookie 12701904178130894609 [ 5090.994139] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5090.995506] LustreError: 191173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5091.097036] Lustre: server umount lustre-MDT0001 complete [ 5102.610691] Lustre: server umount lustre-OST0000 complete [ 5105.778435] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_hostid [ 5109.531712] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 5123.289187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5123.384790] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5123.397105] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5123.436482] Lustre: lustre-MDT0000: new disk, initializing [ 5123.467689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5123.475249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5124.992145] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5129.750365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5129.800578] Lustre: 196574:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5129.831174] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5129.833043] Lustre: Skipped 1 previous similar message [ 5129.835336] LustreError: 197354:0:(llog.c:1696:llog_backup()) MGC192.168.203.138@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 5129.840510] LustreError: 197354:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 5129.844104] Lustre: 197354:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.138@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 5129.854838] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5129.857776] Lustre: Skipped 1 previous similar message [ 5129.895073] Lustre: lustre-MDT0001: new disk, initializing [ 5129.936249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5129.954076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5129.961050] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5131.551793] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5135.751140] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5137.801070] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5140.403029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5140.512795] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5140.515207] Lustre: Skipped 1 previous similar message [ 5140.517340] LustreError: 198517:0:(llog.c:1696:llog_backup()) MGC192.168.203.138@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5140.521240] LustreError: 198517:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 5140.524672] Lustre: 198517:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.138@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 5140.529741] Lustre: 198517:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 5140.541185] Lustre: lustre-OST0000: new disk, initializing [ 5140.543462] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5140.546585] Lustre: 198541:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5140.581842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5142.875030] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5147.033287] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5148.883765] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5150.214116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5150.218488] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5150.258333] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5151.022411] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 5152.838540] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5152.927293] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5155.296841] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5155.298264] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5155.299738] LustreError: Skipped 1 previous similar message [ 5155.300432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5155.304496] Lustre: Skipped 1 previous similar message [ 5155.306276] Lustre: Skipped 2 previous similar messages [ 5159.827581] Lustre: server umount lustre-OST0000 complete [ 5165.537084] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5165.537800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5165.541853] Lustre: Skipped 1 previous similar message [ 5165.546438] Lustre: Skipped 1 previous similar message [ 5167.454964] Lustre: server umount lustre-MDT0000 complete [ 5168.985988] LustreError: 196567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908162 with bad export cookie 12701904178130895309 [ 5168.989992] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5168.991512] LustreError: 196567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5169.125499] Lustre: server umount lustre-MDT0001 complete [ 5172.555561] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 18:56:05 (1788908165) [ 5178.930309] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 5183.148534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5183.325521] LustreError: 201680:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5183.335857] LustreError: 201680:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5183.344186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5184.984252] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5188.577726] LustreError: 201681:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5188.593182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5190.334806] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5191.602196] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5194.341163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5196.944185] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5200.532697] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5200.605432] Lustre: lustre-OST0001: new disk, initializing [ 5200.607763] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5200.611630] Lustre: 203986:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5202.843693] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5202.887528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5202.892421] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5202.925441] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5208.640043] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5214.576687] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5215.289484] Lustre: DEBUG MARKER: ADD OST3 [ 5217.083945] loop: module loaded [ 5217.103690] loop0: detected capacity change from 0 to 204800000 [ 5217.241807] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5217.246321] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5217.252234] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5219.622624] loop0: detected capacity change from 0 to 204800000 [ 5221.725361] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5221.801804] Lustre: lustre-OST0003: new disk, initializing [ 5221.804345] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 5221.807669] Lustre: 205717:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5221.862264] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 5221.865868] Lustre: Skipped 3 previous similar messages [ 5222.416155] Lustre: DEBUG MARKER: STOP OST3 [ 5223.202403] Lustre: Failing over lustre-OST0003 [ 5223.255463] Lustre: server umount lustre-OST0003 complete [ 5223.584595] LustreError: 204356:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5224.312849] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5228.369047] Lustre: DEBUG MARKER: START OST3 again [ 5228.755288] LustreError: 203164:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5228.768035] LustreError: 203164:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5231.090305] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5233.397095] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5236.205428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 5236.210482] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 5236.226262] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 5247.559599] Lustre: Failing over lustre-OST0003 [ 5247.600708] Lustre: server umount lustre-OST0003 complete [ 5251.555747] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5251.557299] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5251.565791] LustreError: 205728:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5251.570732] LustreError: 205728:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5260.767878] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5260.770830] LustreError: Skipped 1 previous similar message [ 5260.772489] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5260.776782] Lustre: Skipped 1 previous similar message [ 5260.778903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5261.801201] LustreError: 203165:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5261.807907] LustreError: 203165:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5266.335393] Lustre: server umount lustre-MDT0000 complete [ 5267.566460] LustreError: 203985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908261 with bad export cookie 12701904178130896177 [ 5267.570092] LustreError: 203985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5267.570540] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5272.034949] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5272.038293] Lustre: Skipped 5 previous similar messages [ 5273.847027] Lustre: server umount lustre-MDT0001 complete [ 5275.194548] Lustre: server umount lustre-OST0000 complete [ 5276.622243] Lustre: server umount lustre-OST0001 complete [ 5278.858788] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_hostid [ 5281.846809] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 5295.376582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5295.464230] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5295.477819] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5295.516389] Lustre: lustre-MDT0000: new disk, initializing [ 5295.537965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5295.539889] Lustre: Skipped 1 previous similar message [ 5295.544943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5296.823625] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5301.042988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5301.085186] Lustre: 211130:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5301.198938] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5302.601294] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5306.364502] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5308.368109] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5310.865164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5310.983414] Lustre: 213099:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5312.572059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5312.576221] Lustre: Skipped 1 previous similar message [ 5312.578409] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5312.610297] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5313.263345] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5317.539598] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5319.276823] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5319.371603] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5321.140463] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5321.237648] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5322.723888] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5322.724213] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5322.725791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5322.731584] Lustre: Skipped 6 previous similar messages [ 5322.734529] LustreError: Skipped 1 previous similar message [ 5324.114770] Lustre: server umount lustre-OST0000 complete [ 5331.813059] Lustre: server umount lustre-MDT0000 complete [ 5332.960890] LustreError: 211140:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5332.967682] LustreError: 211140:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 5333.310387] LustreError: 211123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908327 with bad export cookie 12701904178130898438 [ 5333.313547] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5333.314870] LustreError: 211123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5333.449748] Lustre: server umount lustre-MDT0001 complete [ 5334.869724] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 18:58:47 (1788908327) [ 5340.744153] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_hostid [ 5344.438921] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 5358.236539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5358.324714] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5358.335986] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5358.338503] Lustre: Skipped 3 previous similar messages [ 5358.374422] Lustre: lustre-MDT0000: new disk, initializing [ 5358.375904] Lustre: Skipped 2 previous similar messages [ 5358.404086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5359.774037] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5364.130025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5364.176598] Lustre: 217699:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5364.291886] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5365.775176] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5369.900437] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5372.062586] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5374.600747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5374.714115] Lustre: 219666:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5376.101541] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5376.913789] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5380.819852] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5382.243377] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5382.311570] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5383.756387] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5383.838856] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5386.210677] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5386.211551] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5386.213202] LustreError: Skipped 1 previous similar message [ 5386.217802] Lustre: Skipped 2 previous similar messages [ 5386.219793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5386.223054] Lustre: Skipped 5 previous similar messages [ 5386.661592] Lustre: server umount lustre-OST0000 complete [ 5394.071459] Lustre: server umount lustre-MDT0000 complete [ 5395.307528] LustreError: 217692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908389 with bad export cookie 12701904178130899320 [ 5395.310374] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5395.311723] LustreError: 217692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5395.415492] Lustre: server umount lustre-MDT0001 complete [ 5398.404019] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5398.576165] LustreError: 221879:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5400.100392] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5403.153178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5404.550850] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5406.886749] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5408.609159] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5410.801329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5412.865107] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5415.431768] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5433.825463] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5433.829326] LustreError: Skipped 1 previous similar message [ 5438.425382] Lustre: server umount lustre-OST0000 complete [ 5453.280755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5453.283066] Lustre: Skipped 10 previous similar messages [ 5454.303154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5454.373988] Lustre: server umount lustre-MDT0000 complete [ 5455.646142] LustreError: 221906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908449 with bad export cookie 12701904178130900202 [ 5455.649291] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5455.650450] LustreError: 221906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5455.768855] Lustre: server umount lustre-MDT0001 complete [ 5547.123372] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 5548.328350] Key type lgssc unregistered [ 5548.475112] LNet: 225570:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5548.478131] LNetError: 225570:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5548.487810] LNet: Removed LNI 192.168.203.138@tcp [ 5548.837179] Key type .llcrypt unregistered [ 5548.839078] Key type ._llcrypt unregistered [ 5559.292624] Key type ._llcrypt registered [ 5559.294120] Key type .llcrypt registered [ 5559.337670] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_hostid [ 5566.674405] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 5567.103790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5567.141604] alg: No test for adler32 (adler32-zlib) [ 5568.132704] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 5568.257284] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 5569.863649] Key type lgssc registered [ 5570.398441] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5597.626185] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5597.640055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5598.832253] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5598.851322] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5598.901407] Lustre: lustre-MDT0000: new disk, initializing [ 5598.945939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5598.958443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5602.496204] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5611.738802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5611.936137] Lustre: 228897:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5612.022259] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5612.027565] Lustre: Skipped 1 previous similar message [ 5612.103436] Lustre: lustre-MDT0001: new disk, initializing [ 5612.208575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5612.276319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5612.300865] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5616.699214] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5626.467458] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5631.558148] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5636.825491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5636.993351] Lustre: lustre-OST0000: new disk, initializing [ 5636.997114] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5637.003409] Lustre: 230865:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5637.080182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5640.528220] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5646.565901] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5646.884439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5646.893842] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5646.974854] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5649.477293] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5649.608899] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5652.196825] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5652.348851] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5657.569366] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5657.569584] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5657.582508] LustreError: Skipped 1 previous similar message [ 5657.593185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5657.599126] Lustre: Skipped 1 previous similar message [ 5659.837689] Lustre: server umount lustre-OST0000 complete [ 5662.691906] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5662.695156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5662.699339] Lustre: Skipped 2 previous similar messages [ 5667.808433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5667.813677] Lustre: Skipped 2 previous similar messages [ 5668.229727] Lustre: server umount lustre-MDT0000 complete [ 5670.121161] LustreError: 228889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908664 with bad export cookie 10499601700431052168 [ 5670.125100] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5670.128183] LustreError: 228889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5670.296445] Lustre: server umount lustre-MDT0001 complete [ 5679.251907] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 5685.062594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5685.316990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5687.558389] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5690.340283] LustreError: 233793:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5690.351821] LustreError: 233793:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5692.441942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5692.645404] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5695.049966] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5696.753622] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5700.670086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5700.879518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5704.498485] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5709.451709] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5709.584401] Lustre: lustre-OST0001: new disk, initializing [ 5709.586535] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5709.592263] Lustre: 236103:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5709.899116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5709.905306] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5709.983278] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5713.077452] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5721.105674] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5724.656868] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5731.116649] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 19:05:23 (1788908723) [ 5733.857777] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5733.861400] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5733.867537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5734.403981] Lustre: server umount lustre-MDT0000 complete [ 5736.698987] LustreError: 236102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908730 with bad export cookie 10499601700431053050 [ 5736.705721] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5736.892194] Lustre: server umount lustre-MDT0001 complete [ 5752.287161] Lustre: 227122:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788908730/real 1788908730] req@ffff8925be161180 x1875806584400640/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788908746 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5752.287247] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5752.304047] Lustre: 227122:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5757.471762] Lustre: 227122:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788908735/real 1788908735] req@ffff8925990a8e00 x1875806584401280/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788908751 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5758.564546] Lustre: server umount lustre-OST0000 complete [ 5760.873766] Lustre: server umount lustre-OST0001 complete [ 5764.575438] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_hostid [ 5770.822064] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 5793.133702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5793.354200] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5793.387662] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5793.441287] Lustre: lustre-MDT0000: new disk, initializing [ 5793.504495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5793.509701] Lustre: Skipped 1 previous similar message [ 5793.522757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5795.942919] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5802.636311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5802.711469] Lustre: 240040:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5802.758913] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5802.762231] Lustre: Skipped 1 previous similar message [ 5802.806374] Lustre: lustre-MDT0001: new disk, initializing [ 5802.882316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5802.888224] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5805.431711] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5811.416533] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5814.717908] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5818.540991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5818.743569] Lustre: 242008:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5820.522112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5820.530151] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5820.593322] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5822.503512] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5828.456980] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5831.177994] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5831.322537] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5834.072990] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5834.201119] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5836.260711] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5836.265819] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5836.274694] Lustre: Skipped 1 previous similar message [ 5836.280982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5836.285727] Lustre: Skipped 1 previous similar message [ 5841.473397] Lustre: server umount lustre-OST0000 complete [ 5843.938118] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5843.943128] LustreError: Skipped 1 previous similar message [ 5846.496552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5846.496703] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5846.501027] Lustre: Skipped 3 previous similar messages [ 5846.514664] Lustre: Skipped 2 previous similar messages [ 5849.840815] Lustre: server umount lustre-MDT0000 complete [ 5851.617108] LustreError: 240052:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5851.630534] LustreError: 240052:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5852.029719] LustreError: 240845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908845 with bad export cookie 10499601700431055157 [ 5852.033648] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5852.036133] LustreError: 240845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5852.218556] Lustre: server umount lustre-MDT0001 complete [ 5857.173366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5857.441635] LustreError: 244222:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5857.509331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5857.513986] Lustre: Skipped 2 previous similar messages [ 5860.044400] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5862.883393] LustreError: 244223:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5865.086220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5868.412163] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5873.208960] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5876.580752] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5880.705802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5884.764050] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 5889.276883] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6033.892130] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6033.894860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6033.895261] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6033.901935] Lustre: Skipped 1 previous similar message [ 6033.916313] Lustre: Skipped 1 previous similar message [ 6047.199108] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6047.281334] Lustre: server umount lustre-OST0000 complete [ 6055.364973] Lustre: server umount lustre-MDT0000 complete [ 6057.253959] LustreError: 244201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909051 with bad export cookie 10499601700431056039 [ 6057.258217] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6057.259313] LustreError: 244201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6057.449680] Lustre: server umount lustre-MDT0001 complete [ 6153.696529] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 6154.933038] Key type lgssc unregistered [ 6155.082927] LNet: 247826:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6155.086270] LNetError: 247826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6155.099590] LNet: Removed LNI 192.168.203.138@tcp [ 6155.499157] Key type .llcrypt unregistered [ 6155.500907] Key type ._llcrypt unregistered [ 6166.998658] Key type ._llcrypt registered [ 6167.000373] Key type .llcrypt registered [ 6167.045661] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_hostid [ 6175.011412] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 6175.440654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6175.487794] alg: No test for adler32 (adler32-zlib) [ 6176.395243] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 6176.511378] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 6178.103366] Key type lgssc registered [ 6178.609331] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6197.141601] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 6201.585294] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6201.598790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6202.710234] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6202.752678] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6202.800695] Lustre: lustre-MDT0000: new disk, initializing [ 6202.854273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6202.866406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6204.868655] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 6210.413084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6210.475044] Lustre: 251866:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6210.510302] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6210.513119] Lustre: Skipped 1 previous similar message [ 6210.560748] Lustre: lustre-MDT0001: new disk, initializing [ 6210.611141] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6210.630669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6210.635865] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6212.952774] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 6215.779155] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6218.812493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6218.955138] Lustre: lustre-OST0000: new disk, initializing [ 6218.957184] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6218.960491] Lustre: 253499:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6219.005975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6221.587212] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 6222.668960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6222.674816] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6222.721507] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6226.834326] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6226.913538] Lustre: lustre-OST0001: new disk, initializing [ 6226.915881] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6226.919391] Lustre: 254368:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6226.959257] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6229.551454] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 6230.794535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6230.800647] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6230.842956] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6235.534097] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6239.404051] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6246.371719] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6246.371929] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6246.374889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6246.386577] Lustre: Skipped 2 previous similar messages [ 6249.527794] Lustre: server umount lustre-MDT0000 complete [ 6250.970408] LustreError: 251858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909244 with bad export cookie 14237065295435500732 [ 6250.973338] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6250.979064] LustreError: 251858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6251.122175] Lustre: server umount lustre-MDT0001 complete [ 6262.851333] Lustre: server umount lustre-OST0000 complete [ 6274.606572] Lustre: server umount lustre-OST0001 complete [ 6281.004233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6285.073766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6288.874967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6293.109481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6301.914929] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 6306.690636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6306.702674] Lustre: lustre-MDT0000: reset Object Index mappings [ 6306.991979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6309.167401] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 6312.416146] LustreError: 259371:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6312.423128] LustreError: 259371:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6313.093359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6313.106126] Lustre: lustre-MDT0001: reset Object Index mappings [ 6315.269789] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 6316.834102] Lustre: 260514:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6319.999047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6320.174636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6320.179507] Lustre: Skipped 1 previous similar message [ 6322.871119] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 6323.238082] LustreError: 260866:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6326.842176] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6329.738599] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 6332.145609] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 6332.146736] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6339.077439] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6340.719864] Lustre: 262383:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6342.628375] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6342.629867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6342.641331] Lustre: Skipped 4 previous similar messages [ 6342.647979] Lustre: Skipped 6 previous similar messages [ 6347.745901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6347.750018] Lustre: Skipped 3 previous similar messages [ 6348.712495] Lustre: server umount lustre-MDT0000 complete [ 6350.455518] LustreError: 260116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909344 with bad export cookie 14237065295435507508 [ 6350.458673] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6350.459861] LustreError: 260116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6350.621132] Lustre: server umount lustre-MDT0001 complete [ 6363.099262] Lustre: server umount lustre-OST0000 complete [ 6375.538633] Lustre: server umount lustre-OST0001 complete [ 6384.218312] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 19:16:16 (1788909376) [ 6391.302169] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_hostid [ 6395.393344] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 6412.214588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6412.400246] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6412.421196] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6412.487328] Lustre: lustre-MDT0000: new disk, initializing [ 6412.539748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6412.543865] Lustre: Skipped 1 previous similar message [ 6412.551572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6414.473010] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 6420.102588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6420.159969] Lustre: 266631:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6420.283202] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6422.184499] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 6426.626819] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6428.843974] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6431.859909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6432.002452] Lustre: lustre-OST0000: new disk, initializing [ 6432.005041] Lustre: Skipped 1 previous similar message [ 6432.008630] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6432.011952] Lustre: Skipped 2 previous similar messages [ 6432.015336] Lustre: 268599:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6433.305166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6433.312786] Lustre: Skipped 1 previous similar message [ 6433.318227] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6433.362964] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6435.013214] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 6440.015388] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6442.155700] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6442.256569] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6444.255118] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6444.390338] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6448.610280] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6448.612407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6448.617157] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6448.618615] Lustre: Skipped 1 previous similar message [ 6451.494436] Lustre: server umount lustre-OST0000 complete [ 6453.729575] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6453.736257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6453.739671] Lustre: Skipped 2 previous similar messages [ 6459.504732] Lustre: server umount lustre-MDT0000 complete [ 6461.310389] LustreError: 266623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909455 with bad export cookie 14237065295435509594 [ 6461.313972] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6461.316489] LustreError: 266623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6461.451365] Lustre: server umount lustre-MDT0001 complete [ 6470.077328] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 6475.032723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6475.203430] LustreError: 271529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6475.226241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6475.229035] Lustre: Skipped 2 previous similar messages [ 6476.901645] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 6480.352191] LustreError: 271530:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6480.627155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6482.616684] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 6483.837109] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6486.790159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6489.477245] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 6493.487924] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6493.583460] Lustre: lustre-OST0001: new disk, initializing [ 6493.586375] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6493.590468] Lustre: 273834:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6494.755523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6494.765875] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6494.838581] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6496.363202] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 6502.010968] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6503.753184] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6506.013986] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 6506.049182] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 6511.587140] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6511.589023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6511.589929] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6511.592839] Lustre: Skipped 4 previous similar messages [ 6511.595055] Lustre: Skipped 4 previous similar messages [ 6514.794253] Lustre: server umount lustre-MDT0000 complete [ 6516.363398] LustreError: 273833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909510 with bad export cookie 14237065295435510476 [ 6516.366914] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6516.368498] LustreError: 273833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6516.508316] Lustre: server umount lustre-MDT0001 complete [ 6528.602607] Lustre: server umount lustre-OST0000 complete [ 6539.373201] Lustre: server umount lustre-OST0001 complete [ 6546.717815] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 6551.153741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6551.359821] LustreError: 276883:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6551.398966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6551.403170] Lustre: Skipped 3 previous similar messages [ 6553.231741] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 6556.986759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6558.951386] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 6560.354680] Lustre: 278023:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6563.223136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6565.720488] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 6567.460364] LustreError: 278375:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6567.469540] LustreError: 278375:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6569.217146] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6571.666160] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 6576.625455] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 6576.637284] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 6580.578204] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6581.973967] Lustre: 279903:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6584.052069] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 6584.075860] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 6587.847500] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 19:19:40 (1788909580) [ 6602.214262] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6602.216945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6602.219596] Lustre: Skipped 3 previous similar messages [ 6602.223293] Lustre: Skipped 3 previous similar messages [ 6605.920178] Lustre: server umount lustre-MDT0000 complete [ 6607.334873] LustreError: 276883:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6607.347522] LustreError: 276883:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6607.738291] LustreError: 276865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909601 with bad export cookie 14237065295435516783 [ 6607.742923] LustreError: 276865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6607.743339] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6622.175188] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6622.306335] Lustre: server umount lustre-MDT0001 complete [ 6630.222833] Lustre: server umount lustre-OST0000 complete [ 6646.239079] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6646.340698] Lustre: server umount lustre-OST0001 complete [ 6649.167802] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_hostid [ 6653.854222] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 6675.055740] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 6679.841642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6679.940368] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6679.960775] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6680.000137] Lustre: lustre-MDT0000: new disk, initializing [ 6680.030475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6680.033739] Lustre: Skipped 3 previous similar messages [ 6680.040588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6681.767472] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 6687.292373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6687.355042] Lustre: 284136:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6687.515302] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6689.388178] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 6695.442395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6695.609250] Lustre: 285770:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6697.405278] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6698.586308] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 6704.354934] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6704.449599] Lustre: 286639:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6706.075067] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6707.385924] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 6713.261840] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6715.064185] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6715.067095] Lustre: Skipped 1 previous similar message [ 6721.955497] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 19:21:54 (1788909714) [ 6723.553244] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6723.557661] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6723.565862] Lustre: Skipped 2 previous similar messages [ 6723.569537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6723.577139] Lustre: Skipped 4 previous similar messages [ 6729.143713] Lustre: server umount lustre-MDT0000 complete [ 6730.902394] LustreError: 284930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909724 with bad export cookie 14237065295435524651 [ 6730.904618] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6730.911794] LustreError: 284930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6731.072822] Lustre: server umount lustre-MDT0001 complete [ 6742.169753] Lustre: server umount lustre-OST0000 complete [ 6753.317275] Lustre: server umount lustre-OST0001 complete [ 6757.116211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6757.356407] LustreError: 288880:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6757.368841] LustreError: 288880:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6759.254742] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 6763.012745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6764.935559] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 6767.875330] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6770.160605] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6773.201448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6776.211661] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 6779.483313] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6791.610639] loop1: detected capacity change from 0 to 204800000 [ 6791.741308] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6791.746528] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6791.751637] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6793.828444] loop1: detected capacity change from 0 to 204800000 [ 6796.247907] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6796.292527] LustreError: 288870:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs26320: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 6796.299695] LustreError: 288870:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs26320 [ 6796.302799] LustreError: 288870:0:(mgs_handler.c:517:mgs_target_reg()) fs26320-MDT0001: Error checking target index: rc = -13 [ 6796.307513] LustreError: fs26320-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 6796.312796] LustreError: 292596:0:(tgt_mount.c:2703:server_fill_super()) Unable to start targets: -13 [ 6796.316432] LustreError: 292596:0:(tgt_mount.c:2221:server_put_super()) no obd fs26320-MDT0001 [ 6796.319439] LustreError: 292596:0:(tgt_mount.c:116:server_deregister_mount()) fs26320-MDT0001 not registered [ 6796.338486] Lustre: server umount fs26320-MDT0001 complete [ 6796.340900] LustreError: 292596:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 6799.941152] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6799.974294] Lustre: Setting parameter fs26320-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs26320-MDT0001 [ 6799.995040] Lustre: srv-fs26320-MDT0001: No data found on store. Initialize space. [ 6799.997588] Lustre: Skipped 4 previous similar messages [ 6800.035988] Lustre: fs26320-MDT0001: new disk, initializing [ 6800.038161] Lustre: Skipped 3 previous similar messages [ 6800.081845] Lustre: srv-fs26320-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 6800.085122] Lustre: 293063:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs26320-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6800.089787] LustreError: 293063:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs26320-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6801.755692] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 6805.285690] Lustre: server umount fs26320-MDT0001 complete [ 6810.995946] loop1: detected capacity change from 0 to 204800000 [ 6813.404948] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6813.562452] Lustre: 294628:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs26320-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6813.569569] LustreError: 294628:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs26320-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6815.432295] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 6817.694471] Lustre: server umount fs26320-MDT0001 complete [ 6823.905786] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6823.907845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6823.908333] LustreError: 290847:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6823.908348] LustreError: 290847:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 27 previous similar messages [ 6823.911879] Lustre: Skipped 5 previous similar messages [ 6823.915192] Lustre: Skipped 5 previous similar messages [ 6828.163813] Lustre: server umount lustre-MDT0000 complete [ 6832.179611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6832.252235] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6834.227670] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 6837.734143] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6837.807097] loop1: detected capacity change from 0 to 204800000 [ 6840.467428] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6840.712641] Lustre: 297073:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs26320-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6840.718061] LustreError: 297073:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs26320-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6843.186469] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 6845.888921] Lustre: server umount fs26320-MDT0001 complete [ 6858.210701] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6863.741444] Lustre: server umount lustre-OST0000 complete [ 6868.452832] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6868.462159] LustreError: Skipped 1 previous similar message [ 6871.710265] Lustre: server umount lustre-MDT0000 complete [ 6873.133373] LustreError: 289621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909867 with bad export cookie 14237065295435527332 [ 6873.135897] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6873.138056] LustreError: 289621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6873.274397] Lustre: server umount lustre-MDT0001 complete [ 6877.717400] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 6878.946396] Key type lgssc unregistered [ 6879.109438] LNet: 299412:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6879.115094] LNetError: 299412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6880.169112] LNet: Removed LNI 192.168.203.138@tcp [ 6880.554369] Key type .llcrypt unregistered [ 6880.556106] Key type ._llcrypt unregistered [ 6890.084625] Key type ._llcrypt registered [ 6890.086527] Key type .llcrypt registered [ 6890.135545] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 6890.536554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6890.588322] alg: No test for adler32 (adler32-zlib) [ 6891.469278] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 6891.581925] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 6893.183147] Key type lgssc registered [ 6893.607255] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6896.855432] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6896.863167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6898.040153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6899.700615] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 6903.148032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6903.315701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6904.930779] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 6907.677389] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6910.066709] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6913.005323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6913.161403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6914.212476] LustreError: 302490:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6914.221623] LustreError: 302490:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6915.613206] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 6918.548634] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6919.650608] LustreError: 302491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6922.724780] LustreError: 302643:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6922.736028] LustreError: 302643:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6927.842940] LustreError: 302488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6927.851484] LustreError: 302488:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6931.204603] loop1: detected capacity change from 0 to 204800000 [ 6931.313899] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6931.321950] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6931.332049] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6932.965693] LustreError: 302644:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6932.972503] LustreError: 302644:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6933.468663] loop1: detected capacity change from 0 to 204800000 [ 6935.760447] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6935.833517] Lustre: fs20415-OST0000: new disk, initializing [ 6935.836603] Lustre: srv-fs20415-OST0000: No data found on store. Initialize space. [ 6935.840733] Lustre: 304395:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6935.874747] Lustre: fs20415-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6938.335851] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 6942.165567] Lustre: server umount fs20415-OST0000 complete [ 6943.202597] LustreError: 302488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6943.208957] LustreError: 302488:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6946.818220] loop1: detected capacity change from 0 to 204800000 [ 6948.997550] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6949.078349] Lustre: fs20415-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6951.393412] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 6954.864520] loop2: detected capacity change from 0 to 204800000 [ 6954.946757] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6954.952611] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6954.958595] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6957.016619] loop2: detected capacity change from 0 to 204800000 [ 6959.400704] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6959.446838] LustreError: fs20415-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 6959.453191] LustreError: fs20415-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 6959.458469] LustreError: 307433:0:(tgt_mount.c:2703:server_fill_super()) Unable to start targets: -13 [ 6959.462603] LustreError: 307433:0:(tgt_mount.c:2221:server_put_super()) no obd fs20415-OST0001 [ 6959.466199] LustreError: 307433:0:(tgt_mount.c:116:server_deregister_mount()) fs20415-OST0001 not registered [ 6959.493549] Lustre: server umount fs20415-OST0001 complete [ 6959.495706] LustreError: 307433:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 6959.819273] LustreError: 302643:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6959.831170] LustreError: 302643:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 6964.264928] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6964.339244] Lustre: fs20415-OST0001: new disk, initializing [ 6964.342419] Lustre: srv-fs20415-OST0001: No data found on store. Initialize space. [ 6964.346758] Lustre: 307986:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6964.389870] Lustre: fs20415-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6966.946262] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 6971.282197] Lustre: server umount fs20415-OST0000 complete [ 6975.492717] Lustre: server umount fs20415-OST0001 complete [ 6980.635125] loop1: detected capacity change from 0 to 204800000 [ 6982.917559] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6983.005483] Lustre: fs20415-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6985.740134] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 6988.855912] loop2: detected capacity change from 0 to 204800000 [ 6991.270065] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6993.816902] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 6994.403423] LustreError: 310331:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6994.411214] LustreError: 310331:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [ 6997.585428] Lustre: server umount fs20415-OST0000 complete [ 7001.460248] Lustre: server umount fs20415-OST0001 complete [ 7004.643843] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 7006.688022] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7006.691052] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7006.696604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7009.762985] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7009.764740] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7009.765022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7009.766370] LustreError: Skipped 1 previous similar message [ 7012.057540] Lustre: server umount lustre-OST0000 complete [ 7014.880883] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7014.885546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7014.887706] Lustre: Skipped 2 previous similar messages [ 7019.489262] Lustre: server umount lustre-MDT0000 complete [ 7020.856241] LustreError: 300642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910014 with bad export cookie 15375732517482537803 [ 7020.859112] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7020.861728] LustreError: 300642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7020.974234] Lustre: server umount lustre-MDT0001 complete [ 7024.638554] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 7025.668619] Key type lgssc unregistered [ 7025.792333] LNet: 313899:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7025.795565] LNetError: 313899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7025.804422] LNet: Removed LNI 192.168.203.138@tcp [ 7026.135107] Key type .llcrypt unregistered [ 7026.136247] Key type ._llcrypt unregistered [ 7036.490271] Key type ._llcrypt registered [ 7036.491303] Key type .llcrypt registered [ 7036.528754] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_hostid [ 7042.748324] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 7043.080367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7043.116368] alg: No test for adler32 (adler32-zlib) [ 7043.997472] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 7044.102514] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 7045.687161] Key type lgssc registered [ 7046.123577] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7059.224101] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7059.237330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7060.357146] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7060.371037] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7060.411903] Lustre: lustre-MDT0000: new disk, initializing [ 7060.449960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7060.458172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7062.056750] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 7066.488701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7066.538931] Lustre: 317728:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7066.590480] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7066.593556] Lustre: Skipped 1 previous similar message [ 7066.634444] Lustre: lustre-MDT0001: new disk, initializing [ 7066.674789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7066.694169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7066.699518] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7068.164535] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 7071.848127] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7073.723677] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7076.049439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7076.154530] Lustre: lustre-OST0000: new disk, initializing [ 7076.156332] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7076.160088] Lustre: 319699:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7076.193138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7078.273086] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 7081.981963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7081.986890] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7082.030193] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7082.567677] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7083.519179] Lustre: Mounted lustre-client [ 7083.931423] Lustre: Unmounted lustre-client [ 7088.613183] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7088.614301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7088.619443] Lustre: Skipped 1 previous similar message [ 7088.626212] Lustre: Skipped 1 previous similar message [ 7093.729827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7093.732230] Lustre: Skipped 1 previous similar message [ 7093.788896] Lustre: server umount lustre-OST0000 complete [ 7097.312148] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7097.315199] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7097.320514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7098.848703] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7101.279170] Lustre: server umount lustre-MDT0000 complete [ 7102.641528] LustreError: 317720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910096 with bad export cookie 14473901892947068868 [ 7102.644600] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7102.647420] LustreError: 317720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7102.758767] Lustre: server umount lustre-MDT0001 complete [ 7106.463895] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 7107.647682] Key type lgssc unregistered [ 7136.287566] LNet: 1 local NIs in recovery (showing 1): 192.168.203.138@tcp [ 7136.480159] LNet: 321859:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7136.483795] LNetError: 321859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7214.310354] LNet: Removed LNI 192.168.203.138@tcp [ 7214.671139] Key type .llcrypt unregistered [ 7214.672830] Key type ._llcrypt unregistered [ 7220.468495] Key type ._llcrypt registered [ 7220.469732] Key type .llcrypt registered [ 7220.505600] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_hostid [ 7226.913388] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 7227.244643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7227.284631] alg: No test for adler32 (adler32-zlib) [ 7228.148043] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 7228.233978] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 7229.823183] Key type lgssc registered [ 7230.213926] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7239.770897] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 19:30:32 (1788910232) [ 7241.933685] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7241.940680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7243.012659] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7243.023546] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7243.060333] Lustre: lustre-MDT0000: new disk, initializing [ 7243.082891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7243.090381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7244.475699] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 7248.771944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7248.813225] Lustre: 325054:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7248.836978] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7248.839034] Lustre: Skipped 1 previous similar message [ 7248.871508] Lustre: lustre-MDT0001: new disk, initializing [ 7248.907370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7248.922184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7248.926097] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7250.346535] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 7254.202771] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7256.368224] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7259.092759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7259.208443] Lustre: lustre-OST0000: new disk, initializing [ 7259.211048] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7259.214528] Lustre: 327022:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7259.247368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7259.464331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7259.468266] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7259.506582] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7261.402022] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 7265.290506] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7273.493459] Lustre: server umount lustre-OST0000 complete [ 7274.976125] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7274.981732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7279.583835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7279.587836] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7279.594055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7284.222198] Lustre: server umount lustre-MDT0000 complete [ 7285.709868] LustreError: 325852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910279 with bad export cookie 11564330234343334333 [ 7285.710483] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7285.714708] LustreError: 325852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7296.022437] Lustre: server umount lustre-MDT0001 complete [ 7300.319368] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 7301.485485] Key type lgssc unregistered [ 7301.620413] LNet: 328933:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7301.623310] LNetError: 328933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7301.631404] LNet: Removed LNI 192.168.203.138@tcp [ 7301.993559] Key type .llcrypt unregistered [ 7301.994804] Key type ._llcrypt unregistered [ 7311.054768] Key type ._llcrypt registered [ 7311.056940] Key type .llcrypt registered [ 7311.107645] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 7311.554728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7311.605834] alg: No test for adler32 (adler32-zlib) [ 7312.486414] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 7312.588922] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 7314.183202] Key type lgssc registered [ 7314.670732] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7317.596507] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7317.605139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7318.795057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7320.353498] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 7323.507898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7323.654342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7325.221554] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 7328.174868] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7330.379772] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7333.219081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7333.366287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7335.616363] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 7338.407952] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7352.779684] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7353.376657] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7354.849528] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7354.850229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7354.854370] Lustre: Skipped 1 previous similar message [ 7359.183666] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.38@tcp (stopping) [ 7359.189051] Lustre: Skipped 1 previous similar message [ 7359.456097] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7359.459532] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7364.299493] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.38@tcp (stopping) [ 7364.303738] Lustre: Skipped 3 previous similar messages [ 7369.183266] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7369.243533] Lustre: server umount lustre-MDT0000 complete [ 7369.417660] LustreError: 332672:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7369.423323] LustreError: 332672:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7370.208472] LustreError: 330184:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7370.214388] LustreError: 330184:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7370.623266] LustreError: 332023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910364 with bad export cookie 13348549944016110262 [ 7370.627049] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7370.627573] LustreError: 332023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7374.536307] LustreError: 330180:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7374.545880] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.38@tcp (stopping) [ 7374.549737] Lustre: Skipped 3 previous similar messages [ 7375.328340] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7376.820831] Lustre: server umount lustre-MDT0001 complete [ 7378.239579] Lustre: server umount lustre-OST0000 complete [ 7381.640474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7383.189454] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 7384.268676] Lustre: 334382:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.138@tcp [ 7385.137316] Lustre: server umount MGS complete [ 7388.395205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7388.556496] LustreError: 334952:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7388.565547] LustreError: 334952:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7388.585477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7390.032754] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 7392.963227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7393.108080] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7394.574507] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 7397.781837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7400.302948] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 7403.101235] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7403.718900] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 7405.746955] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7406.276328] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7408.098509] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7408.098710] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7408.101488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7408.105100] Lustre: Skipped 1 previous similar message [ 7408.109392] Lustre: Skipped 2 previous similar messages [ 7409.420457] Lustre: server umount lustre-OST0000 complete [ 7412.828379] Lustre: server umount lustre-MDT0000 complete [ 7413.216539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7413.217166] LustreError: 334947:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7413.222969] Lustre: Skipped 1 previous similar message [ 7414.127270] LustreError: 334934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910408 with bad export cookie 13348549944016111088 [ 7414.130766] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7414.132588] LustreError: 334934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7414.261400] Lustre: server umount lustre-MDT0001 complete [ 7418.149743] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 7419.210830] Key type lgssc unregistered [ 7419.351425] LNet: 338384:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7419.354279] LNetError: 338384:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7419.364299] LNet: Removed LNI 192.168.203.138@tcp [ 7419.665460] Key type .llcrypt unregistered [ 7419.666984] Key type ._llcrypt unregistered [ 7425.981148] Key type ._llcrypt registered [ 7425.982714] Key type .llcrypt registered [ 7426.023371] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_hostid [ 7432.736546] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 7433.107167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7433.154133] alg: No test for adler32 (adler32-zlib) [ 7434.024897] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 7434.121659] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 7435.719157] Key type lgssc registered [ 7436.171622] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7449.547697] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7449.555474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7450.651656] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7450.665756] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7450.707429] Lustre: lustre-MDT0000: new disk, initializing [ 7450.742030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7450.749137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7452.298816] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 7457.024576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7457.078569] Lustre: 341518:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7457.103274] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7457.105832] Lustre: Skipped 1 previous similar message [ 7457.145961] Lustre: lustre-MDT0001: new disk, initializing [ 7457.187609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7457.203133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7457.206868] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7458.886323] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 7462.878615] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7464.967966] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7467.699790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7467.819277] Lustre: lustre-OST0000: new disk, initializing [ 7467.822043] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7467.827326] Lustre: 343484:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7467.866104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7470.080331] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 7474.058315] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7475.664227] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7476.223233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7476.229739] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7476.265924] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7476.774026] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 7478.262251] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7478.345874] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7481.312536] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7481.313137] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7481.315797] LustreError: Skipped 1 previous similar message [ 7481.320862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7481.322683] Lustre: Skipped 1 previous similar message [ 7485.146453] Lustre: server umount lustre-OST0000 complete [ 7487.967943] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7487.971033] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7487.976695] Lustre: Skipped 1 previous similar message [ 7487.979594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7491.552625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7491.553114] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7492.703491] Lustre: server umount lustre-MDT0000 complete [ 7494.021223] LustreError: 341511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910487 with bad export cookie 12216063480331989651 [ 7494.024246] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7494.027440] LustreError: 341511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7494.153202] Lustre: server umount lustre-MDT0001 complete [ 7497.220756] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 19:34:50 (1788910490) [ 7502.768262] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_hostid [ 7506.402509] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 7518.237757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7518.317191] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7518.328046] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7518.363815] Lustre: lustre-MDT0000: new disk, initializing [ 7518.383815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7518.390076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7519.758698] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 7524.029086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7524.066371] Lustre: 348239:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7524.086140] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7524.087952] Lustre: Skipped 1 previous similar message [ 7524.122857] Lustre: lustre-MDT0001: new disk, initializing [ 7524.154604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7524.165322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7524.168184] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7525.599513] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 7529.530054] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7531.351660] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7533.701164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7533.801114] Lustre: lustre-OST0000: new disk, initializing [ 7533.803145] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7533.806481] Lustre: 350206:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7533.833530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7535.354841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7535.358897] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7535.395308] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7535.899376] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 7539.558192] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7550.944894] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7550.945500] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7550.948637] LustreError: Skipped 1 previous similar message [ 7550.955025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7550.958960] Lustre: Skipped 2 previous similar messages [ 7555.675579] Lustre: server umount lustre-OST0000 complete [ 7560.159713] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7560.161789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7560.166070] Lustre: Skipped 1 previous similar message [ 7560.167761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7563.099679] Lustre: server umount lustre-MDT0000 complete [ 7564.439198] LustreError: 349035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910558 with bad export cookie 12216063480331990519 [ 7564.443867] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7564.444967] LustreError: 349035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7564.563437] Lustre: server umount lustre-MDT0001 complete [ 7568.347215] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 7569.411838] Key type lgssc unregistered [ 7569.539437] LNet: 352355:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7569.541720] LNetError: 352355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7569.549867] LNet: Removed LNI 192.168.203.138@tcp [ 7569.842419] Key type .llcrypt unregistered [ 7569.843837] Key type ._llcrypt unregistered [ 7602.646914] Key type ._llcrypt registered [ 7602.648388] Key type .llcrypt registered [ 7602.694152] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 7602.820747] Key type .llcrypt unregistered [ 7602.822242] Key type ._llcrypt unregistered [ 7634.866133] Key type ._llcrypt registered [ 7634.867616] Key type .llcrypt registered [ 7634.907394] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 7635.011324] Key type .llcrypt unregistered [ 7635.012771] Key type ._llcrypt unregistered [ 7647.386233] Key type ._llcrypt registered [ 7647.387455] Key type .llcrypt registered [ 7647.433421] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 7647.552403] Key type .llcrypt unregistered [ 7647.554349] Key type ._llcrypt unregistered [ 7659.065053] Key type ._llcrypt registered [ 7659.066183] Key type .llcrypt registered [ 7659.097043] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 7659.199234] Key type .llcrypt unregistered [ 7659.200544] Key type ._llcrypt unregistered [ 7688.770520] Key type ._llcrypt registered [ 7688.771852] Key type .llcrypt registered [ 7688.819252] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 7688.930299] Key type .llcrypt unregistered [ 7688.932581] Key type ._llcrypt unregistered [ 7701.464893] Key type ._llcrypt registered [ 7701.466142] Key type .llcrypt registered [ 7701.508074] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 7701.620484] Key type .llcrypt unregistered [ 7701.622650] Key type ._llcrypt unregistered [ 7714.151803] Key type ._llcrypt registered [ 7714.153228] Key type .llcrypt registered [ 7714.201926] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 7714.325489] Key type .llcrypt unregistered [ 7714.327264] Key type ._llcrypt unregistered [ 7727.157138] Key type ._llcrypt registered [ 7727.158887] Key type .llcrypt registered [ 7727.208107] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 7727.359560] Key type .llcrypt unregistered [ 7727.362285] Key type ._llcrypt unregistered [ 7751.789699] Key type ._llcrypt registered [ 7751.791761] Key type .llcrypt registered [ 7751.862794] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 7752.085457] Key type .llcrypt unregistered [ 7752.192659] Key type ._llcrypt unregistered [ 7774.206721] Key type ._llcrypt registered [ 7774.208180] Key type .llcrypt registered [ 7774.289639] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 7774.488297] Key type .llcrypt unregistered [ 7774.490239] Key type ._llcrypt unregistered [ 7827.481620] Key type ._llcrypt registered [ 7827.485151] Key type .llcrypt registered [ 7827.583635] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 7827.739506] Key type .llcrypt unregistered [ 7827.743757] Key type ._llcrypt unregistered [ 7845.064294] Key type ._llcrypt registered [ 7845.066548] Key type .llcrypt registered [ 7845.152518] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 7845.353324] Key type .llcrypt unregistered [ 7845.357345] Key type ._llcrypt unregistered [ 7865.809592] Key type ._llcrypt registered [ 7865.811522] Key type .llcrypt registered [ 7865.887613] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 7866.077422] Key type .llcrypt unregistered [ 7866.081701] Key type ._llcrypt unregistered [ 7898.502327] Key type ._llcrypt registered [ 7898.504502] Key type .llcrypt registered [ 7898.567205] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_hostid [ 7907.362094] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 7907.854475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7907.918379] alg: No test for adler32 (adler32-zlib) [ 7908.876398] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 7909.021557] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 7910.639154] Key type lgssc registered [ 7911.176401] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7931.504134] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7931.526543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7932.713505] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 7932.748442] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 7932.811909] Lustre: tst250fs-MDT0000: new disk, initializing [ 7932.875629] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7932.896951] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7935.514638] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 7942.196024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7942.268518] Lustre: 372275:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7942.313988] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 7942.317354] Lustre: Skipped 1 previous similar message [ 7942.390219] Lustre: tst250fs-MDT0001: new disk, initializing [ 7942.458494] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7942.496816] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7942.505542] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7945.513551] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 7948.962995] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7953.445910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7953.643281] Lustre: tst250fs-OST0000: new disk, initializing [ 7953.646262] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 7953.651468] Lustre: 373908:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7953.721886] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7957.855253] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 7961.626093] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7961.637209] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7961.699937] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7965.100266] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7965.225162] Lustre: tst250fs-OST0001: new disk, initializing [ 7965.229282] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 7965.239402] Lustre: 374779:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7965.309687] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7969.409685] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug -1 all [ 7969.851834] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7969.862715] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7969.926062] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7973.248035] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 7988.710485] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7988.718771] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7988.738370] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 7990.753419] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 7990.756286] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7990.766265] Lustre: Skipped 3 previous similar messages [ 7992.842474] Lustre: server umount tst250fs-MDT0000 complete [ 7994.799655] LustreError: 373907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910988 with bad export cookie 102899099398680358 [ 7994.803409] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7994.806591] LustreError: 373907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7994.972424] Lustre: server umount tst250fs-MDT0001 complete [ 8007.536928] Lustre: server umount tst250fs-OST0000 complete [ 8020.111242] Lustre: server umount tst250fs-OST0001 complete [ 8027.914742] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_hostid [ 8031.339456] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 8045.882555] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 19:43:58 (1788911038) [ 8046.558896] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 8053.477667] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_hostid [ 8057.573356] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 8071.784661] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7786 sec ========== 19:44:24 (1788911064) [ 8072.513826] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:44:25 (1788911065) === [ 8074.225695] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:44:27 (1788911067) === [ 8090.842787] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 8092.123205] Key type lgssc unregistered [ 8092.261386] LNet: 383902:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8092.264204] LNetError: 383902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8092.272551] LNet: Removed LNI 192.168.203.138@tcp [ 8092.650122] Key type .llcrypt unregistered [ 8092.651920] Key type ._llcrypt unregistered