[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 1372954291 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001022] APIC: Switch to symmetric I/O mode setup [ 0.002310] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.006806] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008014] pid_max: default: 32768 minimum: 301 [ 0.009172] LSM: Security Framework initializing [ 0.010065] Yama: becoming mindful. [ 0.011047] SELinux: Initializing. [ 0.012079] *** VALIDATE selinux *** [ 0.021751] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026630] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028169] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030089] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031145] *** VALIDATE tmpfs *** [ 0.032645] *** VALIDATE proc *** [ 0.033246] *** VALIDATE cgroup *** [ 0.034012] *** VALIDATE cgroup2 *** [ 0.035295] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036171] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038031] Spectre V2 : User space: Vulnerable [ 0.039008] Speculative Store Bypass: Vulnerable [ 0.042651] debug: unmapping init [mem 0xffffffff9ca59000-0xffffffff9ca60fff] [ 0.044960] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045859] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046030] ... version: 2 [ 0.047011] ... bit width: 48 [ 0.048012] ... generic registers: 4 [ 0.049016] ... value mask: 0000ffffffffffff [ 0.050016] ... max period: 00007fffffffffff [ 0.051013] ... fixed-purpose events: 3 [ 0.052027] ... event mask: 000000070000000f [ 0.053392] rcu: Hierarchical SRCU implementation. [ 0.055810] smp: Bringing up secondary CPUs ... [ 0.056735] x86: Booting SMP configuration: [ 0.057033] .... node #0, CPUs: #1 #2 #3 [ 0.067269] smp: Brought up 1 node, 4 CPUs [ 0.069014] smpboot: Max logical packages: 1 [ 0.070012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.150034] node 0 deferred pages initialised in 78ms [ 0.159140] devtmpfs: initialized [ 0.163000] x86/mm: Memory block size: 128MB [ 0.171390] gcov: version magic: 0x41383552 [ 0.179439] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.188127] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.195379] pinctrl core: initialized pinctrl subsystem [ 0.197244] [ 0.197759] ************************************************************* [ 0.200012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.202012] ** ** [ 0.205012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.207012] ** ** [ 0.209014] ** This means that this kernel is built to expose internal ** [ 0.211016] ** IOMMU data structures, which may compromise security on ** [ 0.214013] ** your system. ** [ 0.215012] ** ** [ 0.218013] ** If you see this message and you are not debugging the ** [ 0.220016] ** kernel, report this immediately to your vendor! ** [ 0.222013] ** ** [ 0.223027] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.226014] ************************************************************* [ 0.228428] NET: Registered protocol family 16 [ 0.230588] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.232058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.235070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.238139] cpuidle: using governor menu [ 0.239997] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.243079] PCI: Using configuration type 1 for base access [ 0.253172] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.288101] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.296131] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.304119] cryptd: max_cpu_qlen set to 1000 [ 0.310761] ACPI: Added _OSI(Module Device) [ 0.315028] ACPI: Added _OSI(Processor Device) [ 0.317017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.320026] ACPI: Added _OSI(Processor Aggregator Device) [ 0.338065] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.353636] ACPI: Interpreter enabled [ 0.359093] ACPI: PM: (supports S0 S3 S4 S5) [ 0.362043] ACPI: Using IOAPIC for interrupt routing [ 0.368076] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.379000] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.416399] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.422056] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.429124] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.438082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.445188] acpiphp: Slot [2] registered [ 0.449157] acpiphp: Slot [5] registered [ 0.453176] acpiphp: Slot [6] registered [ 0.457170] acpiphp: Slot [7] registered [ 0.459094] acpiphp: Slot [8] registered [ 0.465144] acpiphp: Slot [9] registered [ 0.466150] acpiphp: Slot [10] registered [ 0.467116] acpiphp: Slot [3] registered [ 0.469097] acpiphp: Slot [4] registered [ 0.471096] acpiphp: Slot [11] registered [ 0.472092] acpiphp: Slot [12] registered [ 0.474134] acpiphp: Slot [13] registered [ 0.476106] acpiphp: Slot [14] registered [ 0.477120] acpiphp: Slot [15] registered [ 0.479215] acpiphp: Slot [16] registered [ 0.481097] acpiphp: Slot [17] registered [ 0.482212] acpiphp: Slot [18] registered [ 0.484143] acpiphp: Slot [19] registered [ 0.489166] acpiphp: Slot [20] registered [ 0.490000] acpiphp: Slot [21] registered [ 0.490000] acpiphp: Slot [22] registered [ 0.491040] acpiphp: Slot [23] registered [ 0.493092] acpiphp: Slot [24] registered [ 0.494091] acpiphp: Slot [25] registered [ 0.496119] acpiphp: Slot [26] registered [ 0.497197] acpiphp: Slot [27] registered [ 0.499105] acpiphp: Slot [28] registered [ 0.501193] acpiphp: Slot [29] registered [ 0.503117] acpiphp: Slot [30] registered [ 0.506044] acpiphp: Slot [31] registered [ 0.508079] PCI host bridge to bus 0000:00 [ 0.509000] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.511057] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.513079] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.517075] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.520027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.523028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.526348] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.529407] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.533631] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.545969] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.551058] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.554021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.557027] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.563025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.566684] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.569944] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.573049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.575976] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.582021] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.596087] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.603017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.610352] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.619016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.626019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.647020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.657745] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.666020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.673020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.693019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.707132] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.714019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.720024] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.734019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.741000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.749000] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.756022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.774017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.784787] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.794019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.800000] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.819017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.831737] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.840016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.848018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.872018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.887357] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.889866] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.892511] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.895455] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.898233] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.903205] iommu: Default domain type: Passthrough [ 0.905480] SCSI subsystem initialized [ 0.907132] ACPI: bus type USB registered [ 0.908161] usbcore: registered new interface driver usbfs [ 0.911068] usbcore: registered new interface driver hub [ 0.913085] usbcore: registered new device driver usb [ 0.915226] pps_core: LinuxPPS API ver. 1 registered [ 0.917009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.920065] PTP clock support registered [ 0.922369] EDAC MC: Ver: 3.0.0 [ 0.946141] PCI: Using ACPI for IRQ routing [ 0.947000] NetLabel: Initializing [ 0.947000] NetLabel: domain hash size = 128 [ 0.948015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.951112] NetLabel: unlabeled traffic allowed by default [ 0.953335] vgaarb: loaded [ 0.955134] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.957010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.969747] clocksource: Switched to clocksource kvm-clock [ 1.145498] VFS: Disk quotas dquot_6.6.0 [ 1.147172] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.150290] *** VALIDATE ramfs *** [ 1.151986] *** VALIDATE hugetlbfs *** [ 1.156609] pnp: PnP ACPI init [ 1.161737] pnp: PnP ACPI: found 6 devices [ 1.199549] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.204644] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.207353] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.213121] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.219781] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.222886] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.226728] NET: Registered protocol family 2 [ 1.229452] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.234980] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.239397] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.253330] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.258738] TCP: Hash tables configured (established 65536 bind 65536) [ 1.263177] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.268216] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.272603] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.276965] NET: Registered protocol family 1 [ 1.281837] RPC: Registered named UNIX socket transport module. [ 1.285618] RPC: Registered udp transport module. [ 1.287887] RPC: Registered tcp transport module. [ 1.290142] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.293823] NET: Registered protocol family 44 [ 1.296119] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.299324] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.302572] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.306915] PCI: CLS 0 bytes, default 64 [ 1.309783] Unpacking initramfs... [ 3.815433] debug: unmapping init [mem 0xffff9f3d7cc54000-0xffff9f3d7ffbffff] [ 3.824761] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.828742] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.837583] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.971889] Initialise system trusted keyrings [ 4.973859] Key type blacklist registered [ 4.997711] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.008107] zbud: loaded [ 5.011568] *** VALIDATE nfs *** [ 5.012927] *** VALIDATE nfs4 *** [ 5.014629] pstore: using deflate compression [ 5.018543] Platform Keyring initialized [ 5.221580] NET: Registered protocol family 38 [ 5.224857] Key type asymmetric registered [ 5.230752] Asymmetric key parser 'x509' registered [ 5.233497] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.237587] io scheduler mq-deadline registered [ 5.239878] io scheduler kyber registered [ 5.242240] io scheduler bfq registered [ 5.244596] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.248570] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.252507] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.256469] ACPI: Power Button [PWRF] [ 5.265178] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.274465] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.293523] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.302255] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.319949] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.349508] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.387683] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.400145] Non-volatile memory driver v1.3 [ 5.402269] Linux agpgart interface v0.103 [ 5.450842] virtio_blk virtio1: [vda] 145256 512-byte logical blocks (74.4 MB/70.9 MiB) [ 5.459237] vda: detected capacity change from 0 to 74371072 [ 5.483576] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.489760] vdb: detected capacity change from 0 to 1073741824 [ 5.523216] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.529777] vdc: detected capacity change from 0 to 2621440000 [ 5.559538] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.563495] vdd: detected capacity change from 0 to 2621440000 [ 5.581616] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.585574] vde: detected capacity change from 0 to 4294967296 [ 5.601588] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.604781] vdf: detected capacity change from 0 to 4294967296 [ 5.611664] libphy: Fixed MDIO Bus: probed [ 5.618749] usbcore: registered new interface driver usbserial_generic [ 5.622504] usbserial: USB Serial support registered for generic [ 5.625497] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.630836] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.633390] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.636567] mousedev: PS/2 mouse device common for all mice [ 5.641645] rtc_cmos 00:05: RTC can wake from S4 [ 5.645760] rtc_cmos 00:05: registered as rtc0 [ 5.651517] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.655603] intel_pstate: CPU model not supported [ 5.658062] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.664421] hid: raw HID events driver (C) Jiri Kosina [ 5.669237] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.671847] usbcore: registered new interface driver usbhid [ 5.671854] usbhid: USB HID core driver [ 5.671989] drop_monitor: Initializing network drop monitor service [ 5.672136] Initializing XFRM netlink socket [ 5.672592] NET: Registered protocol family 10 [ 5.680867] Segment Routing with IPv6 [ 5.691992] NET: Registered protocol family 17 [ 5.694617] mpls_gso: MPLS GSO support [ 5.697530] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.709613] RAS: Correctable Errors collector initialized. [ 5.712107] AVX version of gcm_enc/dec engaged. [ 5.714077] AES CTR mode by8 optimization enabled [ 5.809589] sched_clock: Marking stable (5809497015, 0)->(7706543628, -1897046613) [ 5.814218] registered taskstats version 1 [ 5.817173] Loading compiled-in X.509 certificates [ 5.819255] zswap: loaded using pool lzo/zbud [ 5.846341] Key type big_key registered [ 5.860134] Key type encrypted registered [ 5.862263] ima: No TPM chip found, activating TPM-bypass! [ 5.864410] ima: Allocated hash algorithm: sha1 [ 5.866284] ima: No architecture policies found [ 5.868103] evm: Initialising EVM extended attributes: [ 5.869861] evm: security.selinux [ 5.871189] evm: security.ima [ 5.872453] evm: security.capability [ 5.873834] evm: HMAC attrs: 0x1 [ 5.876371] rtc_cmos 00:05: setting system clock to 2026-07-09 01:55:54 UTC (1783562154) [ 5.882803] debug: unmapping init [mem 0xffffffff9da03000-0xffffffff9dbfffff] [ 5.885954] debug: unmapping init [mem 0xffffffff9c782000-0xffffffff9ca58fff] [ 5.896339] Write protecting the kernel read-only data: 28672k [ 5.900103] debug: unmapping init [mem 0xffffffff9ae03000-0xffffffff9affffff] [ 5.903086] debug: unmapping init [mem 0xffffffff9b714000-0xffffffff9b7fffff] [ 5.944090] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.956692] systemd[1]: Detected virtualization kvm. [ 5.959441] systemd[1]: Detected architecture x86-64. [ 5.962526] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.003470] systemd[1]: No hostname configured. [ 6.005856] systemd[1]: Set hostname to . [ 6.009797] random: systemd: uninitialized urandom read (16 bytes read) [ 6.014482] systemd[1]: Initializing machine ID from random generator. [ 6.184984] random: ln: uninitialized urandom read (6 bytes read) [ 6.346645] random: systemd: uninitialized urandom read (16 bytes read) [ 6.350562] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 6.357369] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 6.366364] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Reached target Paths. Starting Setup Virtual Console... [ OK ] Reached target Sockets. [ OK ] Reached target Swap. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.023188] device-mapper: uevent: version 1.0.3 [ 8.026663] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 9.208593] virtio_net virtio0 ens2: renamed from eth0 [ 9.517537] random: fast init done [ 9.808127] scsi host0: ata_piix [ 9.855799] scsi host1: ata_piix [ 9.908489] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.912334] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 14.392665] random: crng init done [ 14.395476] random: 7 urandom warning(s) missed due to ratelimiting [ 16.926377] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 18.637632] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 21.430330] printk: systemd: 26 output lines suppressed due to ratelimiting [ 22.421782] SELinux: Disabled at runtime. [ 22.578941] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 22.597903] systemd[1]: Detected virtualization kvm. [ 22.601696] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 24.432478] systemd[1]: initrd-switch-root.service: Succeeded. [ 24.448972] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 24.469464] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 24.486205] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 24.501740] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 24.546842] systemd[1]: Starting Journal Service... Starting Journal Service... [ 24.575650] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached targ[ 24.774981] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS et RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Journal Service. Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 25.811099] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 27.056817] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 27.091500] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 27.566630] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 27.710747] EDAC sbridge: Ver: 1.1.2 [ 30.909353] Key type dns_resolver registered [ 31.679224] NFS: Registering the id_resolver key type [ 31.682564] Key type id_resolver registered [ 31.684819] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg415-server login: [ 71.626692] libcfs: loading out-of-tree module taints kernel. [ 71.649310] Key type ._llcrypt registered [ 71.651018] Key type .llcrypt registered [ 71.698698] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 80.800440] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 81.408039] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 81.415160] alg: No test for adler32 (adler32-zlib) [ 82.456296] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 82.801451] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 84.441053] Key type lgssc registered [ 85.300263] Lustre: Echo OBD driver; http://www.lustre.org/ [ 93.055983] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 113.437327] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 120.017420] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 120.037232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 121.187552] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 121.208372] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 121.262617] Lustre: lustre-MDT0000: new disk, initializing [ 121.331196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 121.345732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 123.397424] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 130.701205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 130.766636] Lustre: 6482:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 130.794509] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 130.797651] Lustre: Skipped 1 previous similar message [ 130.844832] Lustre: lustre-MDT0001: new disk, initializing [ 130.874568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 130.889725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 130.899299] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 132.859454] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 135.851050] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 140.748810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 140.893361] Lustre: lustre-OST0000: new disk, initializing [ 140.897447] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 140.903233] Lustre: 8423:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 140.944056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 144.354276] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 147.982067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 148.005613] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 148.061795] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 154.639502] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 154.703967] Lustre: lustre-OST0001: new disk, initializing [ 154.707296] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 154.711617] Lustre: 9492:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 154.759165] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 157.330147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 157.342733] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 157.393248] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 158.659312] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 167.193980] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 171.181607] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 175.282455] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing check_logdir /tmp/testlogs/ [ 178.142573] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing yml_node [ 180.510796] Lustre: DEBUG MARKER: Client: 2.17.54.105 [ 181.878634] Lustre: DEBUG MARKER: MDS: 2.17.54.105 [ 183.241976] Lustre: DEBUG MARKER: OSS: 2.17.54.105 [ 184.197623] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Jul 8 21:58:50 EDT 2026 [ 193.194358] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 194.041530] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 207.840294] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 207.847908] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 207.859716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 208.865775] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 208.867289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 208.879513] Lustre: Skipped 1 previous similar message [ 208.883516] Lustre: Skipped 1 previous similar message [ 213.986579] LustreError: 7918:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 214.006152] LustreError: 7918:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 214.983492] Lustre: server umount lustre-MDT0000 complete [ 217.107390] LustreError: 6475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783562365 with bad export cookie 4159163829759890421 [ 217.107992] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 217.115251] LustreError: 6475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 219.103808] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 219.103939] LustreError: 6491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 219.104440] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 219.110060] Lustre: Skipped 2 previous similar messages [ 219.120225] LustreError: 6491:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 222.112863] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 222.113813] LustreError: 12119:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 222.119147] Lustre: Skipped 2 previous similar messages [ 222.131959] LustreError: 12119:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 227.234791] LustreError: 6491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 227.236578] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 227.252950] LustreError: 6491:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 227.261396] Lustre: Skipped 1 previous similar message [ 231.391346] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 231.499117] Lustre: server umount lustre-MDT0001 complete [ 233.786985] Lustre: server umount lustre-OST0000 complete [ 236.155191] Lustre: server umount lustre-OST0001 complete [ 240.139703] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 245.310415] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 265.966732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 266.116718] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 266.138980] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 266.192731] Lustre: lustre-MDT0000: new disk, initializing [ 266.245568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 266.256248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 268.978831] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 275.813306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 275.906333] Lustre: 15035:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 275.948250] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 275.952358] Lustre: Skipped 1 previous similar message [ 275.998794] Lustre: lustre-MDT0001: new disk, initializing [ 276.068367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 276.094481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 276.100252] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 278.246438] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 283.240656] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 286.084960] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 289.470446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 289.627780] Lustre: 17003:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 290.731188] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 290.791664] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 292.699166] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 297.811439] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 299.756719] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 299.853700] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 301.694960] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 301.814633] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 306.147222] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 306.148740] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 306.151165] LustreError: Skipped 1 previous similar message [ 306.157944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 306.164315] Lustre: Skipped 1 previous similar message [ 316.895167] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 316.999155] Lustre: server umount lustre-OST0000 complete [ 321.505421] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 321.515759] Lustre: Skipped 1 previous similar message [ 326.624516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 326.630990] Lustre: Skipped 7 previous similar messages [ 333.279161] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 333.413846] Lustre: server umount lustre-MDT0000 complete [ 335.434470] LustreError: 15026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783562484 with bad export cookie 4159163829759891954 [ 335.437976] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 335.444850] LustreError: 15026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 335.626876] Lustre: server umount lustre-MDT0001 complete [ 338.308739] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 22:01:24 (1783562484) [ 348.410290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 348.709678] LustreError: 19889:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 348.762199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 348.768123] Lustre: Skipped 1 previous similar message [ 351.400721] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 355.987664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 358.372779] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 362.130322] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 364.745673] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 365.758062] Lustre: Failing over lustre-MDT0000 [ 365.917447] Lustre: server umount lustre-MDT0000 complete [ 366.560683] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 366.565253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 366.572098] Lustre: Skipped 1 previous similar message [ 366.575734] LustreError: 20617:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 366.586783] LustreError: 20617:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 380.398925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 380.476497] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 381.667667] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 381.681472] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 381.698032] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 382.856604] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 388.668891] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 392.160077] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 392.164826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 392.173704] Lustre: Skipped 1 previous similar message [ 392.177705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 392.181385] Lustre: Skipped 2 previous similar messages [ 397.525114] Lustre: server umount lustre-MDT0000 complete [ 399.762870] LustreError: 19870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783562548 with bad export cookie 4159163829759893368 [ 399.766613] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 399.771891] LustreError: 19870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 399.985143] Lustre: server umount lustre-MDT0001 complete [ 404.641536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 404.907916] LustreError: 23219:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 404.918774] LustreError: 23219:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 404.952467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 404.955724] Lustre: Skipped 2 previous similar messages [ 407.142948] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 411.341810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 413.430471] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 416.967450] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 419.597448] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 420.733814] Lustre: Failing over lustre-MDT0000 [ 420.824121] LustreError: 24789:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffff9f3dfbade900) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 420.859586] Lustre: server umount lustre-MDT0000 complete [ 435.079594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 435.164929] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 437.206969] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 437.345273] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 437.348304] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 437.353184] Lustre: Skipped 1 previous similar message [ 437.360754] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 441.652337] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 447.970316] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 447.970708] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 447.977297] Lustre: Skipped 4 previous similar messages [ 457.186197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 457.192964] Lustre: Skipped 7 previous similar messages [ 458.207792] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 458.288030] Lustre: server umount lustre-MDT0000 complete [ 460.137042] LustreError: 23200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783562608 with bad export cookie 4159163829759894306 [ 460.139048] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 460.143629] LustreError: 23200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 460.311617] Lustre: server umount lustre-MDT0001 complete [ 464.344598] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 22:03:31 (1783562611) [ 471.663405] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 475.876090] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 492.881289] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 497.656730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 497.769813] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 497.788805] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 497.791934] Lustre: Skipped 1 previous similar message [ 497.829866] Lustre: lustre-MDT0000: new disk, initializing [ 497.832210] Lustre: Skipped 1 previous similar message [ 497.867293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 497.870543] Lustre: Skipped 2 previous similar messages [ 497.879593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 497.885382] Lustre: Skipped 1 previous similar message [ 499.776738] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 504.812423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 504.870906] Lustre: 29584:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 505.009178] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 506.862875] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 511.238095] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 513.622358] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 516.443101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 516.587561] Lustre: 31552:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 518.489058] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 518.529590] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 518.542573] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 519.339563] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 524.148476] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 528.664751] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 539.109096] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 539.111435] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 539.117466] Lustre: Skipped 1 previous similar message [ 543.476741] Lustre: server umount lustre-OST0000 complete [ 551.280249] Lustre: server umount lustre-MDT0000 complete [ 552.925579] LustreError: 30378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783562701 with bad export cookie 4159163829759894712 [ 552.929551] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 552.934485] LustreError: 30378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 553.127586] Lustre: server umount lustre-MDT0001 complete [ 557.628615] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 558.852048] Key type lgssc unregistered [ 559.002477] LNet: 33503:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 559.007104] LNetError: 33503:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 559.014521] LNet: Removed LNI 192.168.204.115@tcp [ 559.445278] Key type .llcrypt unregistered [ 559.447462] Key type ._llcrypt unregistered [ 570.394867] Key type ._llcrypt registered [ 570.396324] Key type .llcrypt registered [ 570.441421] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 570.863128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 570.913052] alg: No test for adler32 (adler32-zlib) [ 571.806326] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 571.913921] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 573.511205] Key type lgssc registered [ 573.940335] Lustre: Echo OBD driver; http://www.lustre.org/ [ 577.309433] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 577.319117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 578.527809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 580.237960] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 584.078210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 584.258526] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 585.881966] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 589.046973] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 591.483975] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 594.459473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 594.636166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 597.133089] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 600.476642] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 609.005364] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 616.767470] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 22:06:03 (1783562763) [ 617.708295] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 618.664689] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 621.818934] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 22:06:08 (1783562768) [ 625.655473] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 22:06:12 (1783562772) [ 626.088369] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 629.540841] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 629.545121] Lustre: Skipped 2 previous similar messages [ 630.322353] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 22:06:16 (1783562776) [ 634.076150] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 634.079811] Lustre: Skipped 4 previous similar messages [ 640.282188] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 22:06:26 (1783562786) [ 641.463426] Lustre: *** cfs_fail_loc=131b, val=2*** [ 643.015587] Lustre: *** cfs_fail_loc=131b, val=2*** [ 646.658147] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 22:06:33 (1783562793) [ 647.468395] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 647.471109] Lustre: Skipped 9 previous similar messages [ 652.643790] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 22:06:39 (1783562799) [ 660.450447] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 660.452330] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 660.452763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 660.457737] LustreError: Skipped 1 previous similar message [ 660.462089] Lustre: Skipped 1 previous similar message [ 663.396447] Lustre: server umount lustre-OST0000 complete [ 665.570232] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 665.576856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 665.580820] Lustre: Skipped 2 previous similar messages [ 670.688185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 670.691216] Lustre: Skipped 1 previous similar message [ 675.808340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 675.812016] Lustre: Skipped 1 previous similar message [ 679.391174] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 679.463766] Lustre: server umount lustre-MDT0000 complete [ 680.830276] LustreError: 34873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783562829 with bad export cookie 6202684060584308763 [ 680.833878] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 680.835233] LustreError: 34873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 680.953771] Lustre: server umount lustre-MDT0001 complete [ 685.112331] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 686.388488] Key type lgssc unregistered [ 686.541598] LNet: 42744:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 686.545695] LNetError: 42744:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 686.553556] LNet: Removed LNI 192.168.204.115@tcp [ 686.934184] Key type .llcrypt unregistered [ 686.935842] Key type ._llcrypt unregistered [ 694.526662] Key type ._llcrypt registered [ 694.529871] Key type .llcrypt registered [ 694.580676] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 701.920141] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 702.322181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 702.388056] alg: No test for adler32 (adler32-zlib) [ 703.271526] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 703.378809] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 704.975224] Key type lgssc registered [ 705.482765] Lustre: Echo OBD driver; http://www.lustre.org/ [ 719.700856] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 719.712780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 720.821591] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 720.839504] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 720.888581] Lustre: lustre-MDT0000: new disk, initializing [ 720.926031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 720.934696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 722.711131] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 727.637572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 727.700805] Lustre: 45869:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 727.743889] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 727.747370] Lustre: Skipped 1 previous similar message [ 727.789348] Lustre: lustre-MDT0001: new disk, initializing [ 727.834853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 727.855297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 727.860973] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 729.575742] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 733.832861] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 736.174939] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 739.087314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 739.224448] Lustre: lustre-OST0000: new disk, initializing [ 739.228648] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 739.232314] Lustre: 47849:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 739.280829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 741.780164] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 744.965302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 744.970882] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 745.026654] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 746.341841] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 748.104710] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 748.191477] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 749.885470] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 749.972309] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 755.170576] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 755.171172] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 755.171437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 755.176380] Lustre: Skipped 1 previous similar message [ 755.179866] LustreError: Skipped 1 previous similar message [ 756.893184] Lustre: server umount lustre-OST0000 complete [ 758.752025] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 758.755930] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 758.762297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 758.766852] Lustre: Skipped 1 previous similar message [ 760.294966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 760.295911] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 764.519223] Lustre: server umount lustre-MDT0000 complete [ 766.033553] LustreError: 45860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783562914 with bad export cookie 182811703070034546 [ 766.036665] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 766.038333] LustreError: 45860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 766.175751] Lustre: server umount lustre-MDT0001 complete [ 769.924096] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 22:08:36 (1783562916) [ 772.732206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 772.941410] LustreError: 50306:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 772.949244] LustreError: 50306:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 772.969795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 774.607610] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 778.209183] LustreError: 50307:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 778.262301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 778.446690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 780.133993] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 783.074144] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 785.241917] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 788.175190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 788.343754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 790.834467] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 794.009524] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 801.375691] LustreError: 52830:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 801.812866] Lustre: Modifying parameter general.timeout=1 in log params [ 803.013795] Lustre: Modifying parameter general.timeout=4 in log params [ 803.015760] Lustre: Skipped 2 previous similar messages [ 805.091034] Lustre: Modifying parameter general.timeout=9 in log params [ 805.093984] Lustre: Skipped 4 previous similar messages [ 809.135749] Lustre: Modifying parameter general.timeout=19 in log params [ 809.138483] Lustre: Skipped 9 previous similar messages [ 817.324623] Lustre: Modifying parameter general.timeout=41 in log params [ 817.327451] Lustre: Skipped 21 previous similar messages [ 833.455955] Lustre: Modifying parameter general.timeout=85 in log params [ 833.458950] Lustre: Skipped 43 previous similar messages [ 855.349832] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 22:10:02 (1783563002) [ 856.573721] Lustre: 59735:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 858.048454] Lustre: 59932:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 858.054386] Lustre: 59932:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 860.380452] Lustre: 60227:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 860.389246] Lustre: 60227:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 865.569251] Lustre: Disabling parameter general.ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 in log params [ 865.573705] Lustre: Skipped 60 previous similar messages [ 871.523130] Lustre: 61569:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 871.530625] LustreError: 61569:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 873.296424] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 22:10:19 (1783563019) [ 877.693416] Lustre: server umount lustre-MDT0000 complete [ 879.295952] LustreError: 53768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783563027 with bad export cookie 182811703070035414 [ 879.300264] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 879.301733] LustreError: 53768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 879.307242] LustreError: 50298:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 881.632307] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 881.633093] LustreError: 52261:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 881.638032] Lustre: Skipped 1 previous similar message [ 881.638763] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 881.638767] Lustre: Skipped 1 previous similar message [ 884.640410] LustreError: 50303:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 885.503774] Lustre: server umount lustre-MDT0001 complete [ 905.305299] Lustre: server umount lustre-OST0000 complete [ 916.619639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 916.773351] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 916.891203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 918.488472] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 922.101927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 922.163381] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 922.183285] Lustre: 64097:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 924.198396] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 927.274981] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 929.306437] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 931.637549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 931.716472] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 933.808566] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 936.378573] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 942.229884] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 942.234688] Lustre: Skipped 7 previous similar messages [ 942.255817] Lustre: 66663:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 948.705849] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 948.707389] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 948.710101] LustreError: Skipped 1 previous similar message [ 948.715522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 948.719123] Lustre: Skipped 2 previous similar messages [ 950.484320] Lustre: server umount lustre-OST0000 complete [ 953.311875] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 958.113225] Lustre: server umount lustre-MDT0000 complete [ 958.944939] LustreError: 65034:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 958.954439] LustreError: 65034:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 959.556241] LustreError: 65955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783563108 with bad export cookie 182811703070037787 [ 959.560278] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 959.560968] LustreError: 65955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 959.685368] Lustre: server umount lustre-MDT0001 complete [ 963.500697] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 964.595361] Key type lgssc unregistered [ 964.731382] LNet: 67867:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 964.734283] LNetError: 67867:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 964.745438] LNet: Removed LNI 192.168.204.115@tcp [ 965.086978] Key type .llcrypt unregistered [ 965.089135] Key type ._llcrypt unregistered [ 975.016507] Key type ._llcrypt registered [ 975.018344] Key type .llcrypt registered [ 975.064370] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 975.482957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 975.493589] alg: No test for adler32 (adler32-zlib) [ 976.354366] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 976.453792] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 978.039193] Key type lgssc registered [ 978.458394] Lustre: Echo OBD driver; http://www.lustre.org/ [ 981.346931] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 981.355203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 982.527714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 984.023695] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 987.177818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 987.397330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 988.834867] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 991.395903] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 993.422613] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 995.838169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 996.036353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 998.142716] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1000.937803] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1014.423323] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1018.335834] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1018.338935] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1018.344683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1018.852268] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1018.853795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1018.859080] Lustre: Skipped 1 previous similar message [ 1023.440724] Lustre: server umount lustre-MDT0000 complete [ 1023.968873] LustreError: 69257:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1023.976957] LustreError: 69257:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1024.806361] LustreError: 69996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783563173 with bad export cookie 16798032699164736372 [ 1024.808514] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1024.812532] LustreError: 69996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1024.818383] LustreError: 69249:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 1024.945117] Lustre: server umount lustre-MDT0001 complete [ 1052.743665] Lustre: server umount lustre-OST0000 complete [ 1062.759322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1062.884502] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1062.902486] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1063.016510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1064.614933] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1068.229930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1068.270662] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1068.286176] Lustre: 74144:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1068.476860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1070.137711] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1073.326223] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1075.584351] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1078.511764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1078.608310] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1078.784207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1081.250165] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1083.922190] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1084.539175] Lustre: 76719:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1085.267786] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1091.042193] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1091.044250] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1091.045652] LustreError: Skipped 1 previous similar message [ 1091.052777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1091.056210] Lustre: Skipped 2 previous similar messages [ 1092.901804] Lustre: server umount lustre-OST0000 complete [ 1096.161354] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1096.162523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1096.166914] Lustre: Skipped 2 previous similar messages [ 1100.512649] Lustre: server umount lustre-MDT0000 complete [ 1101.280494] LustreError: 74156:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1101.288519] LustreError: 74156:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1101.960281] LustreError: 76012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783563250 with bad export cookie 16798032699164737170 [ 1101.962054] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1101.966176] LustreError: 76012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1102.098462] Lustre: server umount lustre-MDT0001 complete [ 1106.615692] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 1107.928859] Key type lgssc unregistered [ 1108.085910] LNet: 77925:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1108.091136] LNetError: 77925:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1108.101407] LNet: Removed LNI 192.168.204.115@tcp [ 1108.526237] Key type .llcrypt unregistered [ 1108.528207] Key type ._llcrypt unregistered [ 1119.544076] Key type ._llcrypt registered [ 1119.545596] Key type .llcrypt registered [ 1119.595210] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1120.075955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1120.084826] alg: No test for adler32 (adler32-zlib) [ 1120.962895] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 1121.081615] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 1122.671190] Key type lgssc registered [ 1123.193508] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1126.569389] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1126.580616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1127.803734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1129.643711] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1133.663118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1133.969813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1135.746337] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1138.920843] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1141.305631] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1144.359124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1144.647730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1147.219973] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1150.397645] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1162.900772] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1163.607276] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1163.610541] Lustre: Skipped 1 previous similar message [ 1164.651225] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1164.653808] Lustre: Skipped 2 previous similar messages [ 1166.655818] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1166.659794] Lustre: Skipped 5 previous similar messages [ 1170.909113] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1170.912614] Lustre: Skipped 11 previous similar messages [ 1179.275807] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1179.278579] Lustre: Skipped 22 previous similar messages [ 1195.510776] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1195.513305] Lustre: Skipped 46 previous similar messages [ 1200.614270] Lustre: 87400:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1201.518159] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 22:15:48 (1783563348) [ 1202.061836] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1202.669544] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 22:15:49 (1783563349) [ 1204.191989] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1204.195085] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1204.200069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1205.217544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1205.218557] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1205.223551] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1209.827255] Lustre: server umount lustre-OST0000 complete [ 1215.456417] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1215.457099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1215.461271] Lustre: Skipped 1 previous similar message [ 1215.463838] Lustre: Skipped 1 previous similar message [ 1217.368222] Lustre: server umount lustre-MDT0000 complete [ 1218.696156] LustreError: 81920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783563367 with bad export cookie 1166328884166737727 [ 1218.698959] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1218.701260] LustreError: 81920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1218.822664] Lustre: server umount lustre-MDT0001 complete [ 1222.564352] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 1223.600762] Key type lgssc unregistered [ 1223.737496] LNet: 88648:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1223.741374] LNetError: 88648:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1223.750331] LNet: Removed LNI 192.168.204.115@tcp [ 1224.055257] Key type .llcrypt unregistered [ 1224.056692] Key type ._llcrypt unregistered [ 1225.600384] Key type ._llcrypt registered [ 1225.601753] Key type .llcrypt registered [ 1225.649540] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1227.237489] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules [ 1227.708382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1227.730795] alg: No test for adler32 (adler32-zlib) [ 1228.585397] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 1228.683971] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 1230.271186] Key type lgssc registered [ 1230.743695] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1231.209262] LustreError: 89801:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 1239.286089] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1242.766452] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1252.335319] LustreError: 89801:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1252.347967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1253.548311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1255.120090] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1258.257833] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 22:16:45 (1783563405) [ 1259.975032] LustreError: 90854:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1259.980026] LustreError: 90854:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1259.984082] LustreError: 90854:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 1260.111575] Lustre: server umount lustre-MDT0000 complete [ 1265.523360] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 1266.687931] Key type lgssc unregistered [ 1266.813571] LNet: 92345:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1266.816696] LNetError: 92345:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1266.830431] LNet: Removed LNI 192.168.204.115@tcp [ 1267.185177] Key type .llcrypt unregistered [ 1267.186575] Key type ._llcrypt unregistered [ 1274.717094] Key type ._llcrypt registered [ 1274.718328] Key type .llcrypt registered [ 1274.760455] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1275.171659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1275.181582] alg: No test for adler32 (adler32-zlib) [ 1276.029711] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 1276.123137] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 1277.703174] Key type lgssc registered [ 1278.162627] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1281.017810] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1281.026769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1282.195777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1283.558848] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1286.813341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1287.105126] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1288.847037] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1292.020790] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1294.116538] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1296.880921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1297.145875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1299.339439] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1302.036647] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1332.539546] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 22:17:59 (1783563479) [ 1333.281331] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1334.038783] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 22:18:00 (1783563480) [ 1338.337305] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1338.337362] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1338.339757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1338.351070] Lustre: Skipped 2 previous similar messages [ 1341.351687] Lustre: server umount lustre-MDT0000 complete [ 1342.928494] LustreError: 95388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783563491 with bad export cookie 7236670159976554624 [ 1342.931256] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1342.935730] LustreError: 95388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1343.070830] Lustre: server umount lustre-MDT0001 complete [ 1355.091949] Lustre: server umount lustre-OST0000 complete [ 1359.782379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1360.024480] LustreError: 97807:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1360.035910] LustreError: 97807:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1360.109360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1361.814802] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1365.261930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1365.555847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1367.144270] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1369.988062] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1372.229418] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1377.424160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1377.511597] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1377.517396] LustreError: 97796:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1377.529109] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1377.533336] LustreError: 99890:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 1377.536889] LustreError: 99890:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 1377.540509] LustreError: 99890:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1377.570190] Lustre: server umount lustre-OST0000 complete [ 1377.572709] LustreError: 99890:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1381.124202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1381.220053] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1381.224792] LustreError: 97796:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1381.231341] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1381.237053] LustreError: 100355:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 1381.241676] LustreError: 100355:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 1381.244972] LustreError: 100355:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1381.278817] Lustre: server umount lustre-OST0000 complete [ 1381.281098] LustreError: 100355:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1385.069502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1385.151377] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1385.155072] Lustre: Found index 0 for lustre-OST0000, updating log [ 1385.157812] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1385.185352] Lustre: lustre-OST0000: new disk, initializing [ 1385.187840] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1385.190948] Lustre: 100883:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1385.302974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1388.005978] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1389.540649] Lustre: Failing over lustre-OST0000 [ 1389.610372] Lustre: server umount lustre-OST0000 complete [ 1395.682124] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1395.683711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1395.694589] Lustre: Skipped 1 previous similar message [ 1395.704029] Lustre: Skipped 3 previous similar messages [ 1397.408373] Lustre: server umount lustre-MDT0000 complete [ 1399.228225] LustreError: 97787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783563547 with bad export cookie 7236670159976557354 [ 1399.231275] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1399.233448] LustreError: 97787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1399.391395] Lustre: server umount lustre-MDT0001 complete [ 1403.407943] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 22:19:10 (1783563550) [ 1410.757570] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1415.476788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1415.670302] LustreError: 103223:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1415.710493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1417.463389] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1420.769392] LustreError: 103224:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1421.053658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1423.056654] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1424.359504] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1427.310784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1430.375138] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1433.825030] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1433.861123] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1433.897280] Lustre: lustre-OST0001: new disk, initializing [ 1433.899788] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1433.902893] Lustre: 105539:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1436.329413] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1440.801117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1440.806456] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1440.854098] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1445.276841] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1449.154890] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1450.976314] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 1450.979178] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1450.983433] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1456.731102] Lustre: server umount lustre-MDT0001 complete [ 1457.634325] LustreError: 106014:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1457.641893] LustreError: 106014:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1460.425064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1460.466737] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1460.468981] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1460.487986] Lustre: 103213:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1460.781081] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1460.784506] Lustre: Skipped 3 previous similar messages [ 1462.356913] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1462.442906] LustreError: 107312:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1462.447300] LustreError: 107312:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1462.454329] LustreError: 103215:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1462.493191] LustreError: 107324:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.115@tcp: cfg command failed: rc = -17 [ 1462.497840] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1462.497840] [ 1465.831135] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1466.041966] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 22:20:12 (1783563612) [ 1472.101313] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1479.578964] Lustre: 109146:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1488.696223] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1490.050743] Lustre: 110279:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1542.624088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1542.624433] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1542.624746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1542.624751] Lustre: Skipped 4 previous similar messages [ 1542.630911] Lustre: Skipped 3 previous similar messages [ 1544.100498] Lustre: server umount lustre-MDT0000 complete [ 1545.610656] LustreError: 103203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783563694 with bad export cookie 7236670159976558194 [ 1545.612457] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1545.615050] LustreError: 103203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1545.726380] Lustre: server umount lustre-MDT0001 complete [ 1556.933155] Lustre: server umount lustre-OST0000 complete [ 1567.573258] Lustre: server umount lustre-OST0001 complete [ 1569.514870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1572.985775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1576.110766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1579.589188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1587.157416] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1591.286051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1591.295229] Lustre: lustre-MDT0000: reset Object Index mappings [ 1591.485237] LustreError: 114201:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1591.495747] LustreError: 114201:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1591.522342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1593.246357] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1596.900115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1596.913575] Lustre: lustre-MDT0001: reset Object Index mappings [ 1598.937725] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1600.276573] Lustre: 115348:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1603.206501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1605.804762] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1606.628940] LustreError: 115702:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1606.633800] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 1606.635784] LustreError: 115702:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1608.962552] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1611.088749] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1614.317460] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 1618.411523] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 1618.416930] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 1624.956578] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1626.403176] Lustre: 117246:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1648.224835] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 22:23:14 (1783563794) [ 1679.328406] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1679.334572] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1679.339594] Lustre: Skipped 2 previous similar messages [ 1679.342571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1679.345594] Lustre: Skipped 3 previous similar messages [ 1682.856797] Lustre: server umount lustre-MDT0000 complete [ 1684.297219] LustreError: 114181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783563832 with bad export cookie 7236670159976604996 [ 1684.301717] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1684.303700] LustreError: 114181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1684.960293] LustreError: 114197:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1684.968628] LustreError: 114197:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1690.080873] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1690.083664] Lustre: Skipped 6 previous similar messages [ 1690.574756] Lustre: server umount lustre-MDT0001 complete [ 1698.191427] Lustre: server umount lustre-OST0000 complete [ 1705.904938] Lustre: server umount lustre-OST0001 complete [ 1711.766853] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 1724.654310] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1762.513281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1762.711506] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1762.737429] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1762.829980] Lustre: lustre-MDT0000: new disk, initializing [ 1762.937398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1762.940931] Lustre: Skipped 3 previous similar messages [ 1762.982382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1765.884714] hrtimer: interrupt took 8946484 ns [ 1766.432300] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1771.962734] Lustre: server umount lustre-MDT0000 complete [ 1778.942584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1779.203982] Lustre: Found index 0 for lustre-MDT0000, updating log [ 1779.219540] Lustre: 121506:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1779.231474] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 1782.386683] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1785.817057] Lustre: server umount lustre-MDT0000 complete [ 1792.392188] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 22:25:38 (1783563938) [ 1793.544861] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 1794.699524] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 22:25:41 (1783563941) [ 1805.241077] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1811.693361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1814.672430] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1819.838875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1819.895324] Lustre: 123557:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1819.940704] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1819.946800] Lustre: Skipped 1 previous similar message [ 1819.997875] Lustre: lustre-MDT0001: new disk, initializing [ 1820.088592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1820.094484] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1822.431454] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1825.859272] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1830.158508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1830.327441] Lustre: lustre-OST0000: new disk, initializing [ 1830.331486] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1830.335895] Lustre: 125130:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1831.904211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1831.911178] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1831.995099] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1834.002638] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1840.897210] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1841.001814] Lustre: lustre-OST0001: new disk, initializing [ 1841.008182] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1841.015941] Lustre: 126001:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1842.687972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1842.694056] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1842.756590] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1844.566672] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1850.790429] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1853.013171] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2124.774405] Lustre: 132402:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 2124.780895] Lustre: 132402:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 2124.786430] Lustre: 132402:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 2124.790416] Lustre: 132402:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 2124.795495] Lustre: 132402:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 2124.800281] Lustre: 132402:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2299.166547] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 22:34:05 (1783564445) [ 2313.705631] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2313.707776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2313.719687] Lustre: Skipped 7 previous similar messages [ 2313.729261] Lustre: Skipped 3 previous similar messages [ 2318.326891] LustreError: 132622:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffff9f3def0a7f00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2318.411545] Lustre: server umount lustre-MDT0000 complete [ 2318.818085] LustreError: 123567:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2318.833116] LustreError: 123567:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2320.857124] LustreError: 124294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783564469 with bad export cookie 7236670159976662354 [ 2320.860160] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2320.863302] LustreError: 124294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2323.937050] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2323.947815] Lustre: Skipped 2 previous similar messages [ 2335.199134] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2335.422229] Lustre: server umount lustre-MDT0001 complete [ 2352.095820] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2352.170743] Lustre: server umount lustre-OST0000 complete [ 2360.507195] Lustre: server umount lustre-OST0001 complete [ 2364.758350] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 2370.405275] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2392.292692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2392.423375] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2392.441838] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2392.490944] Lustre: lustre-MDT0000: new disk, initializing [ 2392.541953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2392.546748] Lustre: Skipped 5 previous similar messages [ 2392.560639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2395.035619] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2402.140438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2402.233476] Lustre: 135436:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2402.450689] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2405.220405] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2411.544322] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2415.438694] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2419.316611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2419.460894] Lustre: 137402:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2421.078821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2421.086392] Lustre: Skipped 1 previous similar message [ 2421.088593] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2421.147815] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2422.882077] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2428.471941] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2434.145150] Lustre: lustre-MDD0000: changelog on [ 2435.890591] Lustre: lustre-MDD0001: changelog on [ 3418.259044] Lustre: lustre-MDD0001: changelog off [ 3418.934607] Lustre: 140902:0:(llog_cat.c:984:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 3419.256430] Lustre: lustre-MDD0000: changelog off [ 3424.224219] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3424.228148] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3424.235715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3424.239242] Lustre: Skipped 6 previous similar messages [ 3424.738246] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3424.742234] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3427.172721] Lustre: server umount lustre-OST0000 complete [ 3429.857630] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3429.858877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3429.866664] Lustre: Skipped 1 previous similar message [ 3429.869218] Lustre: Skipped 1 previous similar message [ 3434.920493] Lustre: server umount lustre-MDT0000 complete [ 3434.976536] LustreError: 135447:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3434.984647] LustreError: 135447:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 3436.493254] LustreError: 135428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783565585 with bad export cookie 7236670159977428315 [ 3436.496628] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3436.498531] LustreError: 135428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3436.628747] Lustre: server umount lustre-MDT0001 complete [ 3440.513990] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 3441.601573] Key type lgssc unregistered [ 3441.742383] LNet: 146397:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3441.745286] LNetError: 146397:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3441.752564] LNet: Removed LNI 192.168.204.115@tcp [ 3442.124130] Key type .llcrypt unregistered [ 3442.125849] Key type ._llcrypt unregistered [ 3449.528971] Key type ._llcrypt registered [ 3449.530761] Key type .llcrypt registered [ 3449.578259] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 3456.527413] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3456.832106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3456.843367] alg: No test for adler32 (adler32-zlib) [ 3457.712866] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 3457.818738] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 3459.407146] Key type lgssc registered [ 3459.845092] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3472.732509] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3472.738524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3473.846424] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3473.860981] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3473.900589] Lustre: lustre-MDT0000: new disk, initializing [ 3473.940493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3473.949253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3475.577294] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3480.275283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3480.325822] Lustre: 149571:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3480.356196] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3480.359166] Lustre: Skipped 1 previous similar message [ 3480.399175] Lustre: lustre-MDT0001: new disk, initializing [ 3480.443811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3480.461115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3480.466722] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3482.053353] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3486.203217] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3488.326589] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3490.902036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3491.033872] Lustre: lustre-OST0000: new disk, initializing [ 3491.036481] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3491.040129] Lustre: 151537:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3491.078941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3493.505840] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3495.105331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3495.110162] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3495.148637] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3497.801847] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3500.439785] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_module obdecho/obdecho [ 3501.212559] LustreError: 152541:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 3501.217091] LustreError: 152541:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 3501.908974] LustreError: 152637:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 3510.755514] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3510.756609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3510.760715] Lustre: Skipped 2 previous similar messages [ 3510.765300] Lustre: Skipped 2 previous similar messages [ 3512.550591] Lustre: server umount lustre-MDT0000 complete [ 3514.080585] LustreError: 149563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783565662 with bad export cookie 4511053273496478607 [ 3514.083952] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3514.086819] LustreError: 149563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3514.228104] Lustre: server umount lustre-MDT0001 complete [ 3526.123308] Lustre: server umount lustre-OST0000 complete [ 3534.995865] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 3539.657143] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3555.056826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3555.161642] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3555.175766] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3555.217343] Lustre: lustre-MDT0000: new disk, initializing [ 3555.246549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3555.257428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3556.858688] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3561.975889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3562.029138] Lustre: 156347:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3562.055505] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3562.057652] Lustre: Skipped 1 previous similar message [ 3562.097479] Lustre: lustre-MDT0001: new disk, initializing [ 3562.138902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3562.158023] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3562.163175] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3563.906482] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3568.471420] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3570.568241] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3573.605950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3573.745305] Lustre: lustre-OST0000: new disk, initializing [ 3573.745642] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 3573.748721] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3573.756515] Lustre: 158319:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3573.792856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3576.571248] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3578.876607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3578.881682] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3578.900571] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3581.413204] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3583.394958] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3583.500290] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3585.535977] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3585.644591] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3588.066501] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3588.066685] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3588.072364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3588.078359] Lustre: Skipped 1 previous similar message [ 3592.679336] Lustre: server umount lustre-OST0000 complete [ 3598.306810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3598.307193] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3598.307855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3598.320838] Lustre: Skipped 2 previous similar messages [ 3600.616327] Lustre: server umount lustre-MDT0000 complete [ 3602.322516] LustreError: 157142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783565750 with bad export cookie 4511053273496479566 [ 3602.325214] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3602.329860] LustreError: 157142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3602.457518] Lustre: server umount lustre-MDT0001 complete [ 3604.115989] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 22:55:50 (1783565750) [ 3604.864082] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 3605.635227] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 22:55:52 (1783565752) [ 3609.190743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3609.412664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3611.036498] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3614.526250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3616.460971] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3619.652798] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3621.854854] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3624.532029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3627.139859] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3630.042874] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3640.291531] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3640.292318] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3640.292865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3640.292869] Lustre: Skipped 1 previous similar message [ 3640.297708] Lustre: Skipped 1 previous similar message [ 3643.821132] Lustre: server umount lustre-MDT0000 complete [ 3645.259319] LustreError: 160707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783565793 with bad export cookie 4511053273496480448 [ 3645.263148] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3645.265047] LustreError: 160707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3645.408222] LustreError: 162018:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3645.408538] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3645.414815] LustreError: 162018:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3650.530599] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3650.530607] LustreError: 160722:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3650.532960] Lustre: Skipped 5 previous similar messages [ 3650.542857] LustreError: 160722:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3651.447276] Lustre: server umount lustre-MDT0001 complete [ 3653.988980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3658.950830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3659.057156] LustreError: 164568:0:(llog.c:1655:llog_backup()) MGC192.168.204.115@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 3659.060488] Lustre: 164568:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.115@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 3675.104353] LustreError: 164572:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3675.111968] LustreError: 164572:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3677.421354] LustreError: 164591:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3677.429312] LustreError: 164591:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3681.518331] LustreError: 164574:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3681.527660] LustreError: 164574:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3692.781823] LustreError: 164574:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3692.788177] LustreError: 164574:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 3709.920066] LustreError: 164594:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3709.930942] LustreError: 164594:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 3720.159515] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 3720.162659] Lustre: 164568:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.115@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 3741.934077] LustreError: 164594:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3741.945377] LustreError: 164594:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [ 3760.615817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3760.619367] Lustre: Skipped 2 previous similar messages [ 3762.305725] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3764.707891] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3765.768754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3790.304424] LustreError: 147797:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000a0d125c x1870204088981120/t0(0) o250->MGC192.168.204.115@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3792.165633] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3795.256536] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3795.942913] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3796.005192] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3798.438300] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3803.222133] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 4 sec [ 3811.300103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3811.300143] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3811.302190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3811.302203] Lustre: Skipped 1 previous similar message [ 3811.306291] Lustre: Skipped 2 previous similar messages [ 3813.956968] Lustre: server umount lustre-MDT0000 complete [ 3815.516284] LustreError: 160752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783565964 with bad export cookie 4511053273496486923 [ 3815.519118] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3815.523403] LustreError: 160752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3815.638394] Lustre: server umount lustre-MDT0001 complete [ 3826.529171] Lustre: server umount lustre-OST0000 complete [ 3834.668608] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 3838.321582] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3852.620222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3852.728897] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3852.743774] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3852.784890] Lustre: lustre-MDT0000: new disk, initializing [ 3852.821587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3852.824824] Lustre: Skipped 1 previous similar message [ 3852.831156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3854.484333] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3859.394166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3859.444560] Lustre: 169646:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3859.571497] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3861.164431] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3865.577637] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3867.883469] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3870.746373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3870.866870] Lustre: 171615:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3873.170576] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3876.363442] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3876.410277] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3877.363152] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3879.055655] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3879.143517] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3880.768031] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3880.861742] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3886.562708] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3886.564740] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3886.566622] LustreError: Skipped 1 previous similar message [ 3886.576313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3886.580659] Lustre: Skipped 3 previous similar messages [ 3887.844688] Lustre: server umount lustre-OST0000 complete [ 3895.582825] Lustre: server umount lustre-MDT0000 complete [ 3896.799962] LustreError: 169657:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3896.807474] LustreError: 169657:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 37 previous similar messages [ 3896.935379] LustreError: 169638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783566045 with bad export cookie 4511053273496503730 [ 3896.937391] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3896.941643] LustreError: 169638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3897.070276] Lustre: server umount lustre-MDT0001 complete [ 3900.600264] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 23:00:47 (1783566047) [ 3903.316072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3905.087989] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3908.507594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3910.272281] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3912.947607] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3914.999618] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3917.391478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3919.494820] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3921.977653] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3936.355899] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 23:01:23 (1783566083) [ 3939.296156] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3939.300044] LustreError: Skipped 1 previous similar message [ 3939.302076] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3939.305878] Lustre: Skipped 3 previous similar messages [ 3943.432674] Lustre: server umount lustre-OST0000 complete [ 3950.810595] Lustre: server umount lustre-MDT0000 complete [ 3952.269281] LustreError: 174003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783566100 with bad export cookie 4511053273496504598 [ 3952.273088] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3952.274391] LustreError: 174003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3952.395808] Lustre: server umount lustre-MDT0001 complete [ 3956.069742] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 3957.083378] Key type lgssc unregistered [ 3957.217349] LNet: 177775:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3957.220273] LNetError: 177775:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3957.232335] LNet: Removed LNI 192.168.204.115@tcp [ 3957.534867] Key type .llcrypt unregistered [ 3957.536037] Key type ._llcrypt unregistered [ 3965.553917] Key type ._llcrypt registered [ 3965.555386] Key type .llcrypt registered [ 3965.593881] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3965.928462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3965.986569] alg: No test for adler32 (adler32-zlib) [ 3966.839846] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 3966.934326] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 3968.519378] Key type lgssc registered [ 3968.903736] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3971.614901] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3971.620914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3987.039403] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3992.159470] LustreError: 178994:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.115@tcp: failed processing log, type 4: rc = -110 [ 3997.279507] LustreError: 178994:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 3997.285287] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 3997.288140] LustreError: MGC192.168.204.115@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 3997.295312] LustreError: 178994:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 3997.299138] LustreError: 178994:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 3997.303506] LustreError: 178994:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 3997.327523] Lustre: server umount lustre-OST0000 complete [ 3997.329758] LustreError: 178994:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4000.632305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4001.786028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4003.210952] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4006.243540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4006.377121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4007.850981] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4010.637232] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4012.679607] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4015.171174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4015.249144] LustreError: 181307:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4015.253181] Lustre: 181307:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.115@tcp: invalid local config log lustre-OST0000: rc = -22 [ 4015.305876] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4017.518095] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4019.980554] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4021.730509] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4021.732294] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4021.739663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4021.741964] Lustre: Skipped 2 previous similar messages [ 4026.849548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4026.851564] Lustre: Skipped 2 previous similar messages [ 4031.970229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4031.971990] Lustre: Skipped 2 previous similar messages [ 4035.039103] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4035.095939] Lustre: server umount lustre-MDT0000 complete [ 4036.356082] LustreError: 179473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783566184 with bad export cookie 12755798820034823360 [ 4036.359600] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4036.361301] LustreError: 179473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4036.470135] Lustre: server umount lustre-MDT0001 complete [ 4048.029867] Lustre: server umount lustre-OST0000 complete [ 4051.466465] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 4055.468642] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 4068.971754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4069.056047] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4069.067786] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4069.103064] Lustre: lustre-MDT0000: new disk, initializing [ 4069.130824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4069.138439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4070.705912] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4075.273453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4075.327441] Lustre: 184826:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4075.356692] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4075.359871] Lustre: Skipped 1 previous similar message [ 4075.396931] Lustre: lustre-MDT0001: new disk, initializing [ 4075.435866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4075.450129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4075.454018] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4077.125090] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4081.196445] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4083.263521] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4085.641264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4085.742528] Lustre: lustre-OST0000: new disk, initializing [ 4085.744801] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4085.747623] Lustre: 186796:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4085.773477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4085.911774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4085.916047] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4085.940908] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4087.898601] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4091.770588] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4093.263725] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4093.351906] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4094.883566] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4094.960951] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4096.485791] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4096.486489] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4096.487579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4096.489148] LustreError: Skipped 1 previous similar message [ 4096.502076] Lustre: Skipped 3 previous similar messages [ 4101.603587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4101.606442] Lustre: Skipped 2 previous similar messages [ 4101.781375] Lustre: server umount lustre-OST0000 complete [ 4106.210071] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4106.215169] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4109.469692] Lustre: server umount lustre-MDT0000 complete [ 4110.918105] LustreError: 184817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783566259 with bad export cookie 12755798820034824060 [ 4110.920494] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4110.923203] LustreError: 184817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4111.044453] Lustre: server umount lustre-MDT0001 complete [ 4114.161362] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 23:04:20 (1783566260) [ 4120.870237] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 4121.969561] Key type lgssc unregistered [ 4122.111389] LNet: 189736:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4122.115725] LNetError: 189736:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4122.125301] LNet: Removed LNI 192.168.204.115@tcp [ 4122.460157] Key type .llcrypt unregistered [ 4122.461522] Key type ._llcrypt unregistered [ 4130.418754] Key type ._llcrypt registered [ 4130.419955] Key type .llcrypt registered [ 4130.456729] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 4130.838973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4130.866970] alg: No test for adler32 (adler32-zlib) [ 4131.745482] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 4131.847658] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 4133.447160] Key type lgssc registered [ 4133.846924] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4136.761493] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4136.770487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4137.963192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4139.625466] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4142.707513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4142.843534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4144.319059] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4146.863570] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4148.918554] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4151.708210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4151.779422] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4151.780808] LustreError: 192793:0:(llog.c:1696:llog_backup()) MGC192.168.204.115@tcp: failed to backup log lustre-OST0000: rc = -28 [ 4151.784178] Lustre: 192793:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.115@tcp: can't backup local config lustre-OST0000: rc = -28 [ 4151.825985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4153.860964] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4156.401939] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4157.129637] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4157.131535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4157.134553] Lustre: Skipped 1 previous similar message [ 4162.528081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4162.530351] Lustre: Skipped 4 previous similar messages [ 4163.292091] Lustre: server umount lustre-MDT0000 complete [ 4164.566299] LustreError: 192821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783566313 with bad export cookie 5335007848265209209 [ 4164.568954] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4164.571169] LustreError: 192821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4164.678778] Lustre: server umount lustre-MDT0001 complete [ 4176.282945] Lustre: server umount lustre-OST0000 complete [ 4179.613794] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 4183.167527] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 4195.856205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4195.950316] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4195.961306] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4195.998111] Lustre: lustre-MDT0000: new disk, initializing [ 4196.020316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4196.046341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4197.499148] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4201.702139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4201.743724] Lustre: 196309:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4201.762986] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4201.764460] Lustre: Skipped 1 previous similar message [ 4201.765802] LustreError: 197092:0:(llog.c:1696:llog_backup()) MGC192.168.204.115@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 4201.768872] LustreError: 197092:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 4201.771599] Lustre: 197092:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.115@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 4201.781322] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4201.783398] Lustre: Skipped 1 previous similar message [ 4201.818179] Lustre: lustre-MDT0001: new disk, initializing [ 4201.850241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4201.862846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4201.866546] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4203.262379] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4207.123747] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4209.267472] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4211.913787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4212.030678] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4212.032164] Lustre: Skipped 1 previous similar message [ 4212.033570] LustreError: 198253:0:(llog.c:1696:llog_backup()) MGC192.168.204.115@tcp: failed to backup log lustre-OST0000: rc = -28 [ 4212.037132] LustreError: 198253:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 4212.039657] Lustre: 198253:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.115@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 4212.043174] Lustre: 198253:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 4212.053667] Lustre: lustre-OST0000: new disk, initializing [ 4212.056496] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4212.059719] Lustre: 198277:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4212.098561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4213.047379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4213.051931] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4213.092974] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4214.310720] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4218.482789] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4220.321901] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4220.423296] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4222.089396] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4222.180946] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4223.459441] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4223.459719] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4223.465296] Lustre: Skipped 1 previous similar message [ 4223.470738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4223.473133] Lustre: Skipped 1 previous similar message [ 4228.579715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4228.581947] Lustre: Skipped 1 previous similar message [ 4233.697146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4233.699644] Lustre: Skipped 1 previous similar message [ 4237.279260] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4237.347472] Lustre: server umount lustre-OST0000 complete [ 4238.816119] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4238.820902] Lustre: Skipped 2 previous similar messages [ 4243.936052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4243.938317] Lustre: Skipped 3 previous similar messages [ 4253.151159] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4253.213970] Lustre: server umount lustre-MDT0000 complete [ 4254.176152] LustreError: 196322:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4254.183161] LustreError: 196322:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4254.587347] LustreError: 196303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783566403 with bad export cookie 5335007848265209895 [ 4254.591020] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4254.592141] LustreError: 196303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4254.709355] Lustre: server umount lustre-MDT0001 complete [ 4257.781624] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 23:06:44 (1783566404) [ 4263.653722] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 4267.383019] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4267.525530] LustreError: 201401:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4267.540063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4268.921124] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4271.873891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4273.473153] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4274.454691] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4276.685181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4278.867082] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4281.853671] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4281.918690] Lustre: lustre-OST0001: new disk, initializing [ 4281.920679] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4281.923655] Lustre: 203706:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4284.018959] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4289.540181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4289.544921] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4289.589568] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4289.740172] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4296.506054] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4297.140690] Lustre: DEBUG MARKER: ADD OST3 [ 4298.853655] loop: module loaded [ 4298.870812] loop0: detected capacity change from 0 to 204800000 [ 4298.939836] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4298.944719] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4298.948881] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4300.949362] loop0: detected capacity change from 0 to 204800000 [ 4302.826860] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4302.899606] Lustre: lustre-OST0003: new disk, initializing [ 4302.901799] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 4302.905115] Lustre: 205440:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4302.946661] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 4302.949203] Lustre: Skipped 3 previous similar messages [ 4303.916865] Lustre: DEBUG MARKER: STOP OST3 [ 4304.208353] Lustre: *** cfs_fail_loc=2109, val=2*** [ 4304.238194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 4304.240727] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 4304.250412] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 4304.587209] Lustre: Failing over lustre-OST0003 [ 4304.628458] Lustre: server umount lustre-OST0003 complete [ 4305.092648] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4306.271093] LustreError: 205701:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 4308.985252] Lustre: DEBUG MARKER: START OST3 again [ 4309.476195] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4309.477954] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4309.481574] LustreError: 202883:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4309.484631] LustreError: Skipped 1 previous similar message [ 4311.200873] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4311.300058] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 4312.994376] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4313.014461] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4313.014834] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4313.198897] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4326.521821] Lustre: Failing over lustre-OST0003 [ 4326.561172] Lustre: server umount lustre-OST0003 complete [ 4326.885562] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4326.886662] LustreError: 205450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4326.889217] Lustre: Skipped 2 previous similar messages [ 4326.893867] LustreError: 205450:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4332.004497] LustreError: 202885:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4332.009644] LustreError: 202885:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4340.197187] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4340.198567] LustreError: 202885:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4340.198810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4340.198815] Lustre: Skipped 2 previous similar messages [ 4340.200365] Lustre: Skipped 3 previous similar messages [ 4340.204415] LustreError: 202885:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4345.119961] Lustre: server umount lustre-MDT0000 complete [ 4346.346645] LustreError: 201382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783566494 with bad export cookie 5335007848265210763 [ 4346.350364] LustreError: 201382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4346.351901] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4352.643194] Lustre: server umount lustre-MDT0001 complete [ 4353.966204] Lustre: server umount lustre-OST0000 complete [ 4355.214105] Lustre: server umount lustre-OST0001 complete [ 4357.435854] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 4360.215174] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 4372.018347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4372.097551] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4372.107738] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4372.142744] Lustre: lustre-MDT0000: new disk, initializing [ 4372.170450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4372.173272] Lustre: Skipped 1 previous similar message [ 4372.179113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4373.554182] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4377.587523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4377.623164] Lustre: 210850:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4377.714839] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4378.953954] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4382.422407] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4384.151382] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4386.199981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4386.290297] Lustre: 212818:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4388.161064] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4388.277819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4388.280633] Lustre: Skipped 1 previous similar message [ 4388.282280] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4388.311925] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4391.753514] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4393.193592] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4393.266789] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4394.707116] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4394.773299] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4398.560712] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4398.562673] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4398.565543] Lustre: Skipped 2 previous similar messages [ 4398.567237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4398.569023] Lustre: Skipped 8 previous similar messages [ 4401.485347] Lustre: server umount lustre-OST0000 complete [ 4403.680352] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4403.683091] LustreError: Skipped 1 previous similar message [ 4408.792978] Lustre: server umount lustre-MDT0000 complete [ 4408.799985] LustreError: 211651:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4408.804685] LustreError: 211651:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 4409.984254] LustreError: 210844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783566558 with bad export cookie 5335007848265213045 [ 4409.986965] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4409.989037] LustreError: 210844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4410.104116] Lustre: server umount lustre-MDT0001 complete [ 4411.304972] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 23:09:18 (1783566558) [ 4416.697982] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 4419.925546] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 4431.976308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4432.054990] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4432.065863] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4432.068229] Lustre: Skipped 3 previous similar messages [ 4432.103472] Lustre: lustre-MDT0000: new disk, initializing [ 4432.104755] Lustre: Skipped 2 previous similar messages [ 4432.127220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4433.488056] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4437.463499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4437.504205] Lustre: 217419:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4437.603193] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4438.901485] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4442.462727] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4444.166892] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4446.374540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4446.471426] Lustre: 219385:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4448.372788] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4448.533490] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4451.995888] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4453.320220] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4453.383114] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4454.701523] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4454.778716] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4458.980821] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4458.981297] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4458.984582] LustreError: Skipped 1 previous similar message [ 4458.988602] Lustre: Skipped 3 previous similar messages [ 4461.521538] Lustre: server umount lustre-OST0000 complete [ 4463.072467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4463.074896] Lustre: Skipped 4 previous similar messages [ 4468.883661] Lustre: server umount lustre-MDT0000 complete [ 4469.215898] LustreError: 217429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4469.220748] LustreError: 217429:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4470.118642] LustreError: 217411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783566618 with bad export cookie 5335007848265213927 [ 4470.122488] LustreError: 217411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4470.122502] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4470.232649] Lustre: server umount lustre-MDT0001 complete [ 4473.059142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4474.578301] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4477.519568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4479.085488] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4481.582256] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4483.429592] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4485.683225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4487.726645] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4490.178693] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4508.642689] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4508.645011] LustreError: Skipped 1 previous similar message [ 4512.976709] Lustre: server umount lustre-OST0000 complete [ 4520.407953] Lustre: server umount lustre-MDT0000 complete [ 4521.712455] LustreError: 221578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783566670 with bad export cookie 5335007848265214809 [ 4521.715825] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4521.717531] LustreError: 221578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4521.829833] Lustre: server umount lustre-MDT0001 complete [ 4621.904933] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 4622.912630] Key type lgssc unregistered [ 4623.031330] LNet: 225276:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4623.033805] LNetError: 225276:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4623.041303] LNet: Removed LNI 192.168.204.115@tcp [ 4623.343113] Key type .llcrypt unregistered [ 4623.344921] Key type ._llcrypt unregistered [ 4631.576105] Key type ._llcrypt registered [ 4631.577206] Key type .llcrypt registered [ 4631.617797] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 4638.511416] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 4638.874581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4638.905429] alg: No test for adler32 (adler32-zlib) [ 4639.787081] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 4639.888304] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 4641.479208] Key type lgssc registered [ 4641.929690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4654.404503] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4654.412672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4655.495432] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4655.508755] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4655.544953] Lustre: lustre-MDT0000: new disk, initializing [ 4655.577153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4655.584777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4657.161303] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4662.043220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4662.091266] Lustre: 228613:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4662.132391] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4662.135165] Lustre: Skipped 1 previous similar message [ 4662.174533] Lustre: lustre-MDT0001: new disk, initializing [ 4662.211678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4662.226670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4662.230973] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4663.734126] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4667.448530] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4669.287619] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4671.588795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4671.691057] Lustre: lustre-OST0000: new disk, initializing [ 4671.693118] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4671.697085] Lustre: 230582:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4671.727017] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4673.843896] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4674.365086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4674.369127] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4674.408239] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4677.885855] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4679.518682] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4679.601400] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4681.177919] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4681.267307] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4684.768812] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4684.770651] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4684.777938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4684.780100] Lustre: Skipped 1 previous similar message [ 4688.091159] Lustre: server umount lustre-OST0000 complete [ 4689.888781] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4689.889514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4689.894030] Lustre: Skipped 2 previous similar messages [ 4695.009116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4695.011841] Lustre: Skipped 2 previous similar messages [ 4695.654248] Lustre: server umount lustre-MDT0000 complete [ 4697.143531] LustreError: 229409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783566845 with bad export cookie 2738990087370534357 [ 4697.146592] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4697.148481] LustreError: 229409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4697.263765] Lustre: server umount lustre-MDT0001 complete [ 4704.465635] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 4708.428502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4708.562625] LustreError: 233512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4708.570831] LustreError: 233512:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4708.577420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4709.998260] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4713.025275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4713.151336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4714.632280] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4715.710256] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4718.018034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4720.125272] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4723.131392] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4723.205079] Lustre: lustre-OST0001: new disk, initializing [ 4723.208204] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4723.212808] Lustre: 235817:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4723.220075] Lustre: lustre-OST0001: Not available for connect from 0@lo (not set up) [ 4723.268635] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4723.270978] Lustre: Skipped 1 previous similar message [ 4725.215638] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4728.299616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4728.303215] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4728.318679] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4730.789040] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4737.609466] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4741.440398] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 23:14:48 (1783566888) [ 4743.650806] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4743.652630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4743.655022] Lustre: Skipped 2 previous similar messages [ 4743.656887] Lustre: Skipped 1 previous similar message [ 4748.250813] Lustre: server umount lustre-MDT0000 complete [ 4748.767939] LustreError: 233531:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4748.774610] LustreError: 233531:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4749.487514] LustreError: 234645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783566898 with bad export cookie 2738990087370535239 [ 4749.490783] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4749.491601] LustreError: 234645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4749.611966] Lustre: server umount lustre-MDT0001 complete [ 4761.093333] Lustre: server umount lustre-OST0000 complete [ 4771.558238] Lustre: server umount lustre-OST0001 complete [ 4773.681407] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 4777.094801] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 4789.163556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4789.267188] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4789.278979] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4789.314661] Lustre: lustre-MDT0000: new disk, initializing [ 4789.338518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4789.345705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4790.772457] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4795.091470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4795.132835] Lustre: 239755:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4795.234148] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4796.671593] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4800.862340] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4803.022857] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4805.644818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4805.758244] Lustre: lustre-OST0000: new disk, initializing [ 4805.760053] Lustre: Skipped 1 previous similar message [ 4805.762415] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4805.765235] Lustre: Skipped 2 previous similar messages [ 4805.768680] Lustre: 241724:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4807.238873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4807.243762] Lustre: Skipped 1 previous similar message [ 4807.246729] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4807.292352] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4808.079053] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4812.071587] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4813.574721] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4813.659710] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4815.101603] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4815.186366] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4817.891992] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4817.893363] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4817.895399] LustreError: Skipped 1 previous similar message [ 4817.899786] Lustre: Skipped 1 previous similar message [ 4817.904964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4817.907305] Lustre: Skipped 3 previous similar messages [ 4830.175199] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4830.228993] Lustre: server umount lustre-OST0000 complete [ 4833.248972] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4833.254062] Lustre: Skipped 2 previous similar messages [ 4837.732946] Lustre: server umount lustre-MDT0000 complete [ 4838.368325] LustreError: 240558:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4838.374357] LustreError: 240558:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4839.046149] LustreError: 239748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783566987 with bad export cookie 2738990087370537360 [ 4839.050534] LustreError: 239748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4839.050573] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4839.171205] Lustre: server umount lustre-MDT0001 complete [ 4842.287200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4842.439576] LustreError: 243937:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4842.462442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4842.465214] Lustre: Skipped 2 previous similar messages [ 4843.827133] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4846.872474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4848.564683] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4851.127276] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4853.112557] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4855.573538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4857.857651] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4860.271496] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5005.279829] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5005.282124] LustreError: Skipped 1 previous similar message [ 5005.283903] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5005.287845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5005.290018] Lustre: Skipped 6 previous similar messages [ 5010.446556] Lustre: server umount lustre-OST0000 complete [ 5016.031982] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5016.035208] LustreError: Skipped 1 previous similar message [ 5017.746815] Lustre: server umount lustre-MDT0000 complete [ 5018.959094] LustreError: 244674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783567167 with bad export cookie 2738990087370538242 [ 5018.961986] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5018.962917] LustreError: 244674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5019.087093] Lustre: server umount lustre-MDT0001 complete [ 5134.562439] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 5135.537240] Key type lgssc unregistered [ 5135.660303] LNet: 247546:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5135.664239] LNetError: 247546:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5135.674273] LNet: Removed LNI 192.168.204.115@tcp [ 5135.991226] Key type .llcrypt unregistered [ 5135.993172] Key type ._llcrypt unregistered [ 5143.991754] Key type ._llcrypt registered [ 5143.992649] Key type .llcrypt registered [ 5144.025254] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 5150.233823] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 5150.542598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5150.558692] alg: No test for adler32 (adler32-zlib) [ 5151.420343] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 5151.507093] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 5153.079128] Key type lgssc registered [ 5153.415355] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5167.389795] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 5170.720270] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5170.727618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5171.810635] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5171.821821] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5171.860792] Lustre: lustre-MDT0000: new disk, initializing [ 5171.882187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5171.889116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5173.187473] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5177.313036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5177.358825] Lustre: 251603:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5177.382895] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5177.385599] Lustre: Skipped 1 previous similar message [ 5177.420476] Lustre: lustre-MDT0001: new disk, initializing [ 5177.451315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5177.463902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5177.466866] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5178.770426] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5181.086513] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5183.178802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5183.272662] Lustre: lustre-OST0000: new disk, initializing [ 5183.274705] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5183.277379] Lustre: 253227:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5183.305109] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5184.564061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5184.566623] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5184.612461] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5185.180464] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5189.184327] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5189.252978] Lustre: lustre-OST0001: new disk, initializing [ 5189.254466] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5189.256475] Lustre: 254102:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5189.288050] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5191.166418] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5192.508335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5192.511206] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5192.535771] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5196.441121] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5203.027712] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5208.034689] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5208.035800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5208.038737] Lustre: Skipped 2 previous similar messages [ 5208.040436] Lustre: Skipped 1 previous similar message [ 5211.935479] Lustre: server umount lustre-MDT0000 complete [ 5213.149476] LustreError: 251595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783567361 with bad export cookie 12882648478042969508 [ 5213.152089] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5213.154767] LustreError: 251595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5213.165071] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5213.167348] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5213.170773] Lustre: Skipped 2 previous similar messages [ 5213.173294] Lustre: Skipped 2 previous similar messages [ 5213.276371] Lustre: server umount lustre-MDT0001 complete [ 5220.674420] Lustre: server umount lustre-OST0000 complete [ 5222.037255] Lustre: server umount lustre-OST0001 complete [ 5227.250653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5230.887021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5234.028419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5237.580919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5243.689777] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 5246.958345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5246.966358] Lustre: lustre-MDT0000: reset Object Index mappings [ 5247.097776] LustreError: 259098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5247.105083] LustreError: 259098:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5247.116908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5248.462451] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5251.319664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5251.326380] Lustre: lustre-MDT0001: reset Object Index mappings [ 5252.878733] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5253.900378] Lustre: 260240:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5256.084471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5256.209262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5256.211202] Lustre: Skipped 1 previous similar message [ 5257.250710] LustreError: 260595:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5257.254995] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5258.009970] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5260.708317] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5261.812965] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 5262.751527] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5265.725508] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5266.862172] Lustre: 262106:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5272.037103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5272.041498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5272.041523] Lustre: Skipped 1 previous similar message [ 5272.046197] Lustre: Skipped 2 previous similar messages [ 5272.048302] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5274.137327] Lustre: server umount lustre-MDT0000 complete [ 5275.307969] LustreError: 259081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783567423 with bad export cookie 12882648478042976228 [ 5275.312428] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5275.312990] LustreError: 259081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5275.417377] Lustre: server umount lustre-MDT0001 complete [ 5287.359243] Lustre: server umount lustre-OST0000 complete [ 5298.844875] Lustre: server umount lustre-OST0001 complete [ 5304.672399] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 23:24:11 (1783567451) [ 5309.497058] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 5312.287440] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 5323.337666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5323.434165] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5323.446310] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5323.484547] Lustre: lustre-MDT0000: new disk, initializing [ 5323.507775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5323.510085] Lustre: Skipped 1 previous similar message [ 5323.514766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5324.888356] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5328.833196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5328.869037] Lustre: 266353:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5328.956821] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5330.244274] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5333.836266] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5335.466411] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5337.586924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5337.686816] Lustre: lustre-OST0000: new disk, initializing [ 5337.688354] Lustre: Skipped 1 previous similar message [ 5337.689951] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5337.692704] Lustre: Skipped 2 previous similar messages [ 5337.695275] Lustre: 268322:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5339.634730] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5339.637556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5339.641447] Lustre: Skipped 1 previous similar message [ 5339.644102] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5339.675111] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5343.141425] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5344.529610] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5344.604935] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5345.918671] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5345.999526] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5348.690223] Lustre: server umount lustre-OST0000 complete [ 5354.463813] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5354.467030] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5354.470935] Lustre: Skipped 1 previous similar message [ 5354.472462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5354.474248] Lustre: Skipped 3 previous similar messages [ 5360.096167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5360.099146] Lustre: Skipped 2 previous similar messages [ 5364.191101] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5364.252201] Lustre: server umount lustre-MDT0000 complete [ 5365.215950] LustreError: 267161:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5365.222816] LustreError: 267161:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5365.446433] LustreError: 268321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783567514 with bad export cookie 12882648478042978307 [ 5365.449319] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5365.449758] LustreError: 268321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5376.020599] Lustre: server umount lustre-MDT0001 complete [ 5382.020500] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 5385.532793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5385.677022] LustreError: 271251:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5385.690189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5385.692380] Lustre: Skipped 2 previous similar messages [ 5387.148703] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5390.304943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5391.919481] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5392.952681] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5395.305826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5397.512659] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5400.362441] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5400.425436] Lustre: lustre-OST0001: new disk, initializing [ 5400.428153] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5400.431397] Lustre: 273560:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5401.474628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5401.479461] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5401.519140] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5402.372190] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5406.966743] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5408.146279] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5410.113644] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 5410.121451] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 5415.904138] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5415.908366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5415.913139] Lustre: Skipped 1 previous similar message [ 5415.915519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5418.204811] Lustre: server umount lustre-MDT0000 complete [ 5419.495269] LustreError: 273559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783567568 with bad export cookie 12882648478042979189 [ 5419.498823] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5419.499414] LustreError: 273559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5419.608274] Lustre: server umount lustre-MDT0001 complete [ 5431.240324] Lustre: server umount lustre-OST0000 complete [ 5442.982125] Lustre: server umount lustre-OST0001 complete [ 5448.662728] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 5452.406730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5452.536571] LustreError: 276608:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5452.554354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5452.557523] Lustre: Skipped 3 previous similar messages [ 5453.991825] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5457.051205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5458.698435] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5459.648865] Lustre: 277751:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5461.870160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5463.011179] LustreError: 278103:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5463.018703] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 5464.032952] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5466.988762] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5468.147963] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 5469.104489] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5472.238417] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5473.414202] Lustre: 279624:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5475.272215] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 5475.276797] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 5478.512880] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 23:27:05 (1783567625) [ 5493.215875] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5493.218178] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5493.221599] Lustre: Skipped 3 previous similar messages [ 5493.223503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5493.224942] Lustre: Skipped 4 previous similar messages [ 5496.089146] Lustre: server umount lustre-MDT0000 complete [ 5497.288545] LustreError: 276591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783567645 with bad export cookie 12882648478042985496 [ 5497.291974] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5497.293544] LustreError: 276591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5498.847991] LustreError: 276603:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5498.852403] LustreError: 276603:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5503.477809] Lustre: server umount lustre-MDT0001 complete [ 5519.327133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5519.367522] Lustre: server umount lustre-OST0000 complete [ 5526.611888] Lustre: server umount lustre-OST0001 complete [ 5528.671231] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 5531.864898] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 5547.520147] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 5551.073273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5551.155384] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5551.166836] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5551.200258] Lustre: lustre-MDT0000: new disk, initializing [ 5551.232173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5552.623487] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5557.013061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5557.067306] Lustre: 283853:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5557.184392] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5558.665899] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5563.295273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5563.400626] Lustre: 285487:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5563.853841] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5565.416493] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5569.542383] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5571.559514] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5571.615466] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5575.978634] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5581.029539] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 23:28:47 (1783567727) [ 5581.793296] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5581.794916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5581.798261] Lustre: Skipped 8 previous similar messages [ 5581.799842] Lustre: Skipped 9 previous similar messages [ 5587.807205] Lustre: server umount lustre-MDT0000 complete [ 5589.015493] LustreError: 284649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783567737 with bad export cookie 12882648478042993378 [ 5589.018684] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5589.021033] LustreError: 284649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5589.142637] Lustre: server umount lustre-MDT0001 complete [ 5600.262770] Lustre: server umount lustre-OST0000 complete [ 5611.736983] Lustre: server umount lustre-OST0001 complete [ 5614.664449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5614.811976] LustreError: 288603:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5614.819186] LustreError: 288603:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5614.840571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5614.842823] Lustre: Skipped 7 previous similar messages [ 5616.224508] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5618.983199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5620.417867] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5622.685915] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5624.395507] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5626.529432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5628.656897] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5631.015980] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5647.516361] loop1: detected capacity change from 0 to 204800000 [ 5647.618817] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5647.623043] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5647.628018] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5649.323636] loop1: detected capacity change from 0 to 204800000 [ 5651.238892] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5651.269924] LustreError: 288592:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs809: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 5651.275754] LustreError: 288592:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs809 [ 5651.277480] LustreError: 288592:0:(mgs_handler.c:517:mgs_target_reg()) fs809-MDT0001: Error checking target index: rc = -13 [ 5651.280112] LustreError: fs809-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 5651.282831] LustreError: 292342:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 5651.284858] LustreError: 292342:0:(tgt_mount.c:2129:server_put_super()) no obd fs809-MDT0001 [ 5651.286771] LustreError: 292342:0:(tgt_mount.c:116:server_deregister_mount()) fs809-MDT0001 not registered [ 5651.296540] Lustre: server umount fs809-MDT0001 complete [ 5651.298281] LustreError: 292342:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 5654.284821] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5654.314712] Lustre: Setting parameter fs809-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs809-MDT0001 [ 5654.319960] Lustre: Skipped 2 previous similar messages [ 5654.337540] Lustre: srv-fs809-MDT0001: No data found on store. Initialize space. [ 5654.339898] Lustre: Skipped 4 previous similar messages [ 5654.377897] Lustre: fs809-MDT0001: new disk, initializing [ 5654.379626] Lustre: Skipped 3 previous similar messages [ 5654.413497] Lustre: srv-fs809-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 5654.416396] Lustre: 292808:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs809-MDT0001-mdtlov: can't create new llog: rc = -115 [ 5654.419624] LustreError: 292808:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs809-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 5655.909915] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5658.821151] Lustre: server umount fs809-MDT0001 complete [ 5662.982789] loop1: detected capacity change from 0 to 204800000 [ 5664.786599] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5664.894988] Lustre: 294374:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs809-MDT0001-mdtlov: can't create new llog: rc = -115 [ 5664.898259] LustreError: 294374:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs809-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 5666.160854] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5667.722471] Lustre: server umount fs809-MDT0001 complete [ 5670.880405] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5670.881856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5670.884528] Lustre: Skipped 1 previous similar message [ 5670.886339] Lustre: Skipped 5 previous similar messages [ 5675.487656] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5681.121921] LustreError: 290803:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5681.128488] LustreError: 290803:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 25 previous similar messages [ 5685.215186] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5685.278442] Lustre: server umount lustre-MDT0000 complete [ 5688.108297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5688.147968] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5689.614418] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5692.238596] loop1: detected capacity change from 0 to 204800000 [ 5693.415911] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5694.201226] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5694.339432] Lustre: 296816:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs809-MDT0001-mdtlov: can't create new llog: rc = -115 [ 5694.343419] LustreError: 296816:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs809-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 5695.615253] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5697.151326] Lustre: server umount fs809-MDT0001 complete [ 5703.650911] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5709.394838] Lustre: server umount lustre-OST0000 complete [ 5716.691643] Lustre: server umount lustre-MDT0000 complete [ 5717.818133] LustreError: 288585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783567866 with bad export cookie 12882648478042996052 [ 5717.822607] LustreError: 288585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5717.931272] Lustre: server umount lustre-MDT0001 complete [ 5721.211143] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 5722.141232] Key type lgssc unregistered [ 5722.253382] LNet: 299154:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5722.255416] LNetError: 299154:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5722.266294] LNet: Removed LNI 192.168.204.115@tcp [ 5722.552087] Key type .llcrypt unregistered [ 5722.553110] Key type ._llcrypt unregistered [ 5729.777207] Key type ._llcrypt registered [ 5729.778552] Key type .llcrypt registered [ 5729.816428] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 5730.168766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5730.197739] alg: No test for adler32 (adler32-zlib) [ 5731.058879] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 5731.154055] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 5732.743161] Key type lgssc registered [ 5733.147694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5735.701423] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5735.708823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5736.847743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5738.149587] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5740.812392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5740.943169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5742.196890] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5744.435923] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5746.169639] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5748.374310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5748.498713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5750.352948] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5750.565727] LustreError: 302620:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5750.573952] LustreError: 302620:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5752.583870] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5755.873859] LustreError: 302620:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5760.994857] LustreError: 302226:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5763.316860] LustreError: 302228:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5763.324930] LustreError: 302228:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5768.431336] LustreError: 302226:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5768.438199] LustreError: 302226:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5769.380803] loop1: detected capacity change from 0 to 204800000 [ 5769.436918] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5769.441643] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5769.446577] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5771.128291] loop1: detected capacity change from 0 to 204800000 [ 5772.830357] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5772.880806] Lustre: fs13476-OST0000: new disk, initializing [ 5772.882529] Lustre: srv-fs13476-OST0000: No data found on store. Initialize space. [ 5772.885227] Lustre: 304155:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5772.911336] Lustre: fs13476-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5774.763198] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5777.383710] LustreError: 302620:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5777.388439] LustreError: 302620:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5778.203733] Lustre: server umount fs13476-OST0000 complete [ 5782.135618] loop1: detected capacity change from 0 to 204800000 [ 5784.005657] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5784.061977] Lustre: fs13476-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5785.863497] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5788.629439] loop2: detected capacity change from 0 to 204800000 [ 5788.707929] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5788.713236] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5788.717528] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5790.353986] loop2: detected capacity change from 0 to 204800000 [ 5792.168043] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5792.196270] LustreError: fs13476-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 5792.199598] LustreError: fs13476-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 5792.202829] LustreError: 307190:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 5792.205209] LustreError: 307190:0:(tgt_mount.c:2129:server_put_super()) no obd fs13476-OST0001 [ 5792.207072] LustreError: 307190:0:(tgt_mount.c:116:server_deregister_mount()) fs13476-OST0001 not registered [ 5792.217064] Lustre: server umount fs13476-OST0001 complete [ 5792.218842] LustreError: 307190:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 5794.031183] LustreError: 302228:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5794.037248] LustreError: 302228:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 5795.612504] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5795.655744] Lustre: fs13476-OST0001: new disk, initializing [ 5795.657903] Lustre: srv-fs13476-OST0001: No data found on store. Initialize space. [ 5795.660246] Lustre: 307741:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5795.679520] Lustre: fs13476-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5797.538500] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5800.983200] Lustre: server umount fs13476-OST0000 complete [ 5803.868290] Lustre: server umount fs13476-OST0001 complete [ 5807.596942] loop1: detected capacity change from 0 to 204800000 [ 5809.367678] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5811.142696] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5813.161551] loop2: detected capacity change from 0 to 204800000 [ 5814.932176] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5815.004183] Lustre: fs13476-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5815.007225] Lustre: Skipped 1 previous similar message [ 5816.834956] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5819.572209] Lustre: server umount fs13476-OST0000 complete [ 5822.345118] Lustre: server umount fs13476-OST0001 complete [ 5827.552232] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5827.555264] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5827.561165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5828.577707] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5828.578290] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5828.580922] LustreError: 302229:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5828.580938] LustreError: 302229:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 17 previous similar messages [ 5828.581382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5828.595936] Lustre: Skipped 1 previous similar message [ 5833.696166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5833.698621] Lustre: Skipped 1 previous similar message [ 5838.818045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5838.820757] Lustre: Skipped 1 previous similar message [ 5838.820942] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 5840.351081] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5840.401818] Lustre: server umount lustre-OST0000 complete [ 5843.423976] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5843.426889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5843.432593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5847.705971] Lustre: server umount lustre-MDT0000 complete [ 5848.883122] LustreError: 300383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783567997 with bad export cookie 10569425203425532992 [ 5848.885965] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5848.886481] LustreError: 300383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5848.996673] Lustre: server umount lustre-MDT0001 complete [ 5852.515784] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 5853.521223] Key type lgssc unregistered [ 5853.652348] LNet: 313649:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5853.654762] LNetError: 313649:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5853.664436] LNet: Removed LNI 192.168.204.115@tcp [ 5853.952104] Key type .llcrypt unregistered [ 5853.953682] Key type ._llcrypt unregistered [ 5863.432864] Key type ._llcrypt registered [ 5863.434193] Key type .llcrypt registered [ 5863.465815] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 5869.488204] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 5869.800444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5869.806301] alg: No test for adler32 (adler32-zlib) [ 5870.663574] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 5870.762514] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 5872.343106] Key type lgssc registered [ 5872.701059] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5884.625730] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5884.630898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5885.707674] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5885.717984] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5885.749684] Lustre: lustre-MDT0000: new disk, initializing [ 5885.769544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5885.775058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5887.049413] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5891.052595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5891.090151] Lustre: 317479:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5891.110629] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5891.113640] Lustre: Skipped 1 previous similar message [ 5891.144727] Lustre: lustre-MDT0001: new disk, initializing [ 5891.171996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5891.182835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5891.186806] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5892.463091] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5895.925083] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5897.754919] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5900.042558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5900.149773] Lustre: lustre-OST0000: new disk, initializing [ 5900.151962] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5900.154914] Lustre: 319445:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5900.183468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5902.069035] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5903.478244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5903.481842] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5903.511259] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5905.819698] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5906.528339] Lustre: Mounted lustre-client [ 5906.859277] Lustre: Unmounted lustre-client [ 5911.523208] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5911.528363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5911.530479] Lustre: Skipped 1 previous similar message [ 5915.985068] Lustre: server umount lustre-OST0000 complete [ 5921.761664] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5921.762073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5921.762984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5921.769588] Lustre: Skipped 2 previous similar messages [ 5923.416499] Lustre: server umount lustre-MDT0000 complete [ 5924.672185] LustreError: 317471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783568073 with bad export cookie 12181714060334700626 [ 5924.675891] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5924.678037] LustreError: 317471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5924.802056] Lustre: server umount lustre-MDT0001 complete [ 5928.498948] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 5929.412386] Key type lgssc unregistered [ 5962.911274] LNet: 1 local NIs in recovery (showing 1): 192.168.204.115@tcp [ 5963.295387] LNet: 321605:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5963.298173] LNetError: 321605:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6039.077362] LNet: Removed LNI 192.168.204.115@tcp [ 6039.403177] Key type .llcrypt unregistered [ 6039.405052] Key type ._llcrypt unregistered [ 6045.634105] Key type ._llcrypt registered [ 6045.635188] Key type .llcrypt registered [ 6045.673618] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 6052.435827] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6052.802486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6052.871667] alg: No test for adler32 (adler32-zlib) [ 6053.724375] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 6053.811672] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 6055.391143] Key type lgssc registered [ 6055.787686] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6065.747864] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 23:36:52 (1783568212) [ 6068.016770] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6068.026354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6069.109477] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6069.119721] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6069.161219] Lustre: lustre-MDT0000: new disk, initializing [ 6069.182450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6069.188213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6070.510872] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6074.676056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6074.729543] Lustre: 324802:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6074.757981] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6074.759782] Lustre: Skipped 1 previous similar message [ 6074.799746] Lustre: lustre-MDT0001: new disk, initializing [ 6074.838463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6074.853035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6074.856701] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6076.275492] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6079.958780] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6081.782708] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6084.000450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6084.103865] Lustre: lustre-OST0000: new disk, initializing [ 6084.105631] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6084.109628] Lustre: 326772:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6084.143478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6084.350166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6084.353081] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6084.394584] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6086.164172] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6089.882626] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6099.936681] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6099.938186] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6099.939867] LustreError: Skipped 1 previous similar message [ 6099.945887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6099.949898] Lustre: Skipped 1 previous similar message [ 6101.652330] Lustre: server umount lustre-OST0000 complete [ 6105.056872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6105.057358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6105.060793] Lustre: Skipped 2 previous similar messages [ 6110.176048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6110.178009] Lustre: Skipped 2 previous similar messages [ 6115.295984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6115.298174] Lustre: Skipped 1 previous similar message [ 6117.343163] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6117.402767] Lustre: server umount lustre-MDT0000 complete [ 6118.643222] LustreError: 324794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783568267 with bad export cookie 4565570187794872267 [ 6118.646833] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6118.648066] LustreError: 324794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6118.757444] Lustre: server umount lustre-MDT0001 complete [ 6122.450524] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6123.548545] Key type lgssc unregistered [ 6123.676427] LNet: 328675:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6123.678843] LNetError: 328675:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6123.687271] LNet: Removed LNI 192.168.204.115@tcp [ 6124.042474] Key type .llcrypt unregistered [ 6124.043572] Key type ._llcrypt unregistered [ 6131.990929] Key type ._llcrypt registered [ 6131.991996] Key type .llcrypt registered [ 6132.037118] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6132.441988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6132.494082] alg: No test for adler32 (adler32-zlib) [ 6133.355509] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 6133.456051] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 6135.047166] Key type lgssc registered [ 6135.465486] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6138.175604] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6138.182228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6139.351400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6140.750234] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6143.598506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6143.720746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6145.028925] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6147.380771] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6149.196248] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6151.505047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6151.633894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6153.489774] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6155.799790] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6169.462171] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6170.024971] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6171.380392] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.15@tcp (stopping) [ 6173.400196] Lustre: server umount lustre-MDT0000 complete [ 6174.177403] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6174.178837] LustreError: 329920:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6174.181165] Lustre: Skipped 2 previous similar messages [ 6174.185544] LustreError: 329920:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6174.681244] LustreError: 331765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783568323 with bad export cookie 9257013721802450462 [ 6174.683844] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6174.684809] LustreError: 331765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6176.493701] LustreError: 329922:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6176.502444] LustreError: 329922:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6176.503065] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.15@tcp (stopping) [ 6179.295822] LustreError: 332136:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6179.296178] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6179.307452] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6180.839551] Lustre: server umount lustre-MDT0001 complete [ 6188.176618] Lustre: server umount lustre-OST0000 complete [ 6190.940310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6192.249944] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6193.245504] Lustre: 334123:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.115@tcp [ 6194.049194] Lustre: server umount MGS complete [ 6196.976316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6197.139383] LustreError: 334693:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6197.160870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6198.424695] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6201.154305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6201.264368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6202.589323] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6205.438145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6207.464656] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6209.828054] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6210.403310] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 6212.180875] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6212.703721] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6221.284451] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6221.286625] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6221.287271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6221.292275] Lustre: Skipped 1 previous similar message [ 6223.565272] Lustre: server umount lustre-OST0000 complete [ 6226.400877] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6226.406693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6226.408693] Lustre: Skipped 2 previous similar messages [ 6236.640041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6236.642038] Lustre: Skipped 3 previous similar messages [ 6239.199198] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6239.259787] Lustre: server umount lustre-MDT0000 complete [ 6240.491246] LustreError: 334673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783568389 with bad export cookie 9257013721802451288 [ 6240.494865] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6240.495538] LustreError: 334673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6240.627735] Lustre: server umount lustre-MDT0001 complete [ 6244.231191] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6245.300242] Key type lgssc unregistered [ 6245.442371] LNet: 338125:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6245.444665] LNetError: 338125:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6245.452368] LNet: Removed LNI 192.168.204.115@tcp [ 6245.798123] Key type .llcrypt unregistered [ 6245.800233] Key type ._llcrypt unregistered [ 6251.789587] Key type ._llcrypt registered [ 6251.791420] Key type .llcrypt registered [ 6251.828625] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 6258.122246] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6258.535619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6258.549836] alg: No test for adler32 (adler32-zlib) [ 6259.401748] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 6259.488973] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 6261.071152] Key type lgssc registered [ 6261.446927] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6273.147077] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6273.154473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6274.236251] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6274.247002] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6274.281593] Lustre: lustre-MDT0000: new disk, initializing [ 6274.302499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6274.308130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6275.658519] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6279.882634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6279.922421] Lustre: 341258:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6279.949506] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6279.952636] Lustre: Skipped 1 previous similar message [ 6279.991394] Lustre: lustre-MDT0001: new disk, initializing [ 6280.025636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6280.037589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6280.041592] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6281.477090] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6285.201290] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6287.024745] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6289.366818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6289.474930] Lustre: lustre-OST0000: new disk, initializing [ 6289.477766] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6289.489148] Lustre: 343226:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6289.517445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6291.523965] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6295.312736] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6296.830529] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6299.127032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6299.130861] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6299.161206] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6299.972024] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 6301.446384] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6301.523351] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6304.226267] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6304.227130] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6304.229791] LustreError: Skipped 1 previous similar message [ 6304.234092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6304.237765] Lustre: Skipped 1 previous similar message [ 6308.241516] Lustre: server umount lustre-OST0000 complete [ 6310.879957] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6310.882414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6310.886280] Lustre: Skipped 1 previous similar message [ 6310.887891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6314.464181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6314.464609] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6315.612324] Lustre: server umount lustre-MDT0000 complete [ 6316.876737] LustreError: 341251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783568465 with bad export cookie 9305579931739701779 [ 6316.880588] LustreError: 341251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6316.882652] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6316.992161] Lustre: server umount lustre-MDT0001 complete [ 6320.036991] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 23:41:06 (1783568466) [ 6325.469584] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 6328.490825] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6340.589983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6340.673600] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6340.682476] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6340.718204] Lustre: lustre-MDT0000: new disk, initializing [ 6340.739203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6340.744869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6342.141236] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6346.367499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6346.407627] Lustre: 347994:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6346.433682] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6346.435622] Lustre: Skipped 1 previous similar message [ 6346.472991] Lustre: lustre-MDT0001: new disk, initializing [ 6346.507784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6346.520084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6346.523194] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6347.906307] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6351.639675] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6353.505373] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6355.840636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6355.967103] Lustre: lustre-OST0000: new disk, initializing [ 6355.970131] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6355.974174] Lustre: 349965:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6356.008732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6358.005939] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6361.676578] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6365.685372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6365.689907] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6365.720098] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6378.395741] Lustre: server umount lustre-OST0000 complete [ 6381.025168] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6381.025827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6381.029386] Lustre: Skipped 1 previous similar message [ 6381.033704] Lustre: Skipped 2 previous similar messages [ 6386.144246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6386.147706] Lustre: Skipped 1 previous similar message [ 6390.265131] Lustre: server umount lustre-MDT0000 complete [ 6391.460433] LustreError: 347987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783568540 with bad export cookie 9305579931739702647 [ 6391.463579] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6391.464211] LustreError: 347987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6402.067519] Lustre: server umount lustre-MDT0001 complete [ 6405.821562] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6406.890933] Key type lgssc unregistered [ 6407.029372] LNet: 352116:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6407.032473] LNetError: 352116:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6407.044342] LNet: Removed LNI 192.168.204.115@tcp [ 6407.357832] Key type .llcrypt unregistered [ 6407.358935] Key type ._llcrypt unregistered [ 6439.364811] Key type ._llcrypt registered [ 6439.365879] Key type .llcrypt registered [ 6439.404187] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6439.506420] Key type .llcrypt unregistered [ 6439.507532] Key type ._llcrypt unregistered [ 6467.011487] Key type ._llcrypt registered [ 6467.012583] Key type .llcrypt registered [ 6467.054291] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6467.177284] Key type .llcrypt unregistered [ 6467.179129] Key type ._llcrypt unregistered [ 6478.185551] Key type ._llcrypt registered [ 6478.186714] Key type .llcrypt registered [ 6478.221892] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6478.315350] Key type .llcrypt unregistered [ 6478.316352] Key type ._llcrypt unregistered [ 6488.935772] Key type ._llcrypt registered [ 6488.936890] Key type .llcrypt registered [ 6488.970633] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6489.065320] Key type .llcrypt unregistered [ 6489.066630] Key type ._llcrypt unregistered [ 6515.471351] Key type ._llcrypt registered [ 6515.472399] Key type .llcrypt registered [ 6515.510248] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6515.598259] Key type .llcrypt unregistered [ 6515.599430] Key type ._llcrypt unregistered [ 6526.654565] Key type ._llcrypt registered [ 6526.655580] Key type .llcrypt registered [ 6526.688900] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6526.783181] Key type .llcrypt unregistered [ 6526.784567] Key type ._llcrypt unregistered [ 6537.630610] Key type ._llcrypt registered [ 6537.631649] Key type .llcrypt registered [ 6537.664824] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6537.764188] Key type .llcrypt unregistered [ 6537.765161] Key type ._llcrypt unregistered [ 6548.179883] Key type ._llcrypt registered [ 6548.180903] Key type .llcrypt registered [ 6548.216353] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6548.301221] Key type .llcrypt unregistered [ 6548.302245] Key type ._llcrypt unregistered [ 6559.692365] Key type ._llcrypt registered [ 6559.694045] Key type .llcrypt registered [ 6559.737446] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6559.833299] Key type .llcrypt unregistered [ 6559.834695] Key type ._llcrypt unregistered [ 6570.603201] Key type ._llcrypt registered [ 6570.605013] Key type .llcrypt registered [ 6570.644509] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6570.734338] Key type .llcrypt unregistered [ 6570.736172] Key type ._llcrypt unregistered [ 6591.898585] Key type ._llcrypt registered [ 6591.899763] Key type .llcrypt registered [ 6591.937611] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6592.036527] Key type .llcrypt unregistered [ 6592.037673] Key type ._llcrypt unregistered [ 6603.365802] Key type ._llcrypt registered [ 6603.366783] Key type .llcrypt registered [ 6603.405082] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6603.510343] Key type .llcrypt unregistered [ 6603.511573] Key type ._llcrypt unregistered [ 6614.840384] Key type ._llcrypt registered [ 6614.841589] Key type .llcrypt registered [ 6614.877553] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6614.975204] Key type .llcrypt unregistered [ 6614.976367] Key type ._llcrypt unregistered [ 6633.735195] Key type ._llcrypt registered [ 6633.736272] Key type .llcrypt registered [ 6633.771215] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 6640.274974] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6640.667044] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6640.702598] alg: No test for adler32 (adler32-zlib) [ 6641.558502] Lustre: Lustre: Build Version: 2.17.54_105_g7dbd668 [ 6641.648484] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 6643.231119] Key type lgssc registered [ 6643.606572] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6655.042349] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6655.047851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6656.133599] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 6656.143953] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 6656.179965] Lustre: tst250fs-MDT0000: new disk, initializing [ 6656.201870] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6656.209764] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6657.665703] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6661.871789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6661.910484] Lustre: 372013:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6661.932566] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 6661.934531] Lustre: Skipped 1 previous similar message [ 6661.968861] Lustre: tst250fs-MDT0001: new disk, initializing [ 6661.997786] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6662.010206] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6662.013235] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6663.307345] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6665.601719] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6667.839974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6667.955191] Lustre: tst250fs-OST0000: new disk, initializing [ 6667.956896] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 6667.959668] Lustre: 373647:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6667.996941] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6669.179242] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6669.183932] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6669.232771] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6669.931773] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6674.105902] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6674.173902] Lustre: tst250fs-OST0001: new disk, initializing [ 6674.175692] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 6674.178250] Lustre: 374520:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6674.212787] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6676.125282] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6678.210939] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6678.216136] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6678.250417] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6678.672903] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 6692.831941] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6692.834912] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6692.839101] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 6693.859217] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 6693.859944] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6693.865356] Lustre: Skipped 2 previous similar messages [ 6693.869526] Lustre: Skipped 2 previous similar messages [ 6694.997185] Lustre: server umount tst250fs-MDT0000 complete [ 6696.267242] LustreError: 372809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783568844 with bad export cookie 4580805497254923681 [ 6696.270899] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6696.272843] LustreError: 372809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6696.410142] Lustre: server umount tst250fs-MDT0001 complete [ 6708.230993] Lustre: server umount tst250fs-OST0000 complete [ 6719.773964] Lustre: server umount tst250fs-OST0001 complete [ 6725.808661] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 6728.546280] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6739.038907] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 23:48:05 (1783568885) [ 6739.588519] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 6745.085552] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 6748.588756] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6759.453983] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6575 sec ========== 23:48:26 (1783568906) [ 6760.048522] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:48:26 (1783568906) === [ 6761.221875] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:48:28 (1783568908) === [ 6774.746867] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6775.809953] Key type lgssc unregistered [ 6775.946395] LNet: 383629:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6775.950153] LNetError: 383629:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6775.959277] LNet: Removed LNI 192.168.204.115@tcp [ 6776.283094] Key type .llcrypt unregistered [ 6776.284311] Key type ._llcrypt unregistered