[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 480419030 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001008] APIC: Switch to symmetric I/O mode setup [ 0.002270] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007760] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009008] pid_max: default: 32768 minimum: 301 [ 0.011019] LSM: Security Framework initializing [ 0.012062] Yama: becoming mindful. [ 0.013044] SELinux: Initializing. [ 0.014071] *** VALIDATE selinux *** [ 0.023188] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028350] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029144] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030114] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031119] *** VALIDATE tmpfs *** [ 0.033234] *** VALIDATE proc *** [ 0.034244] *** VALIDATE cgroup *** [ 0.036006] *** VALIDATE cgroup2 *** [ 0.037267] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039148] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041022] Spectre V2 : User space: Vulnerable [ 0.042007] Speculative Store Bypass: Vulnerable [ 0.045380] debug: unmapping init [mem 0xffffffff98659000-0xffffffff98660fff] [ 0.047165] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048672] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049021] ... version: 2 [ 0.050015] ... bit width: 48 [ 0.051014] ... generic registers: 4 [ 0.052014] ... value mask: 0000ffffffffffff [ 0.053015] ... max period: 00007fffffffffff [ 0.054015] ... fixed-purpose events: 3 [ 0.055012] ... event mask: 000000070000000f [ 0.057215] rcu: Hierarchical SRCU implementation. [ 0.059450] smp: Bringing up secondary CPUs ... [ 0.060588] x86: Booting SMP configuration: [ 0.061041] .... node #0, CPUs: #1 #2 #3 [ 0.064675] smp: Brought up 1 node, 4 CPUs [ 0.066016] smpboot: Max logical packages: 1 [ 0.067017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.240723] node 0 deferred pages initialised in 170ms [ 0.244485] devtmpfs: initialized [ 0.245246] x86/mm: Memory block size: 128MB [ 0.247736] gcov: version magic: 0x41383552 [ 0.248682] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.253122] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.256432] pinctrl core: initialized pinctrl subsystem [ 0.258242] [ 0.258891] ************************************************************* [ 0.262018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.265015] ** ** [ 0.267016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.269020] ** ** [ 0.271016] ** This means that this kernel is built to expose internal ** [ 0.274014] ** IOMMU data structures, which may compromise security on ** [ 0.277019] ** your system. ** [ 0.279015] ** ** [ 0.282019] ** If you see this message and you are not debugging the ** [ 0.284016] ** kernel, report this immediately to your vendor! ** [ 0.287014] ** ** [ 0.290014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.292011] ************************************************************* [ 0.293842] NET: Registered protocol family 16 [ 0.296484] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.298070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.301088] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.305143] cpuidle: using governor menu [ 0.306796] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.310291] PCI: Using configuration type 1 for base access [ 0.312152] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.322148] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.325034] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.329184] cryptd: max_cpu_qlen set to 1000 [ 0.333568] ACPI: Added _OSI(Module Device) [ 0.335021] ACPI: Added _OSI(Processor Device) [ 0.337019] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.338017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.342788] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.350053] ACPI: Interpreter enabled [ 0.352080] ACPI: PM: (supports S0 S3 S4 S5) [ 0.353013] ACPI: Using IOAPIC for interrupt routing [ 0.355177] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.359528] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.370910] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.373056] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.375021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.377115] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.383641] acpiphp: Slot [2] registered [ 0.385109] acpiphp: Slot [5] registered [ 0.386181] acpiphp: Slot [6] registered [ 0.388206] acpiphp: Slot [7] registered [ 0.390243] acpiphp: Slot [8] registered [ 0.392199] acpiphp: Slot [9] registered [ 0.393168] acpiphp: Slot [10] registered [ 0.395168] acpiphp: Slot [3] registered [ 0.397159] acpiphp: Slot [4] registered [ 0.398150] acpiphp: Slot [11] registered [ 0.400203] acpiphp: Slot [12] registered [ 0.402144] acpiphp: Slot [13] registered [ 0.405146] acpiphp: Slot [14] registered [ 0.406101] acpiphp: Slot [15] registered [ 0.407114] acpiphp: Slot [16] registered [ 0.408135] acpiphp: Slot [17] registered [ 0.410116] acpiphp: Slot [18] registered [ 0.411089] acpiphp: Slot [19] registered [ 0.412199] acpiphp: Slot [20] registered [ 0.414127] acpiphp: Slot [21] registered [ 0.415180] acpiphp: Slot [22] registered [ 0.417180] acpiphp: Slot [23] registered [ 0.419125] acpiphp: Slot [24] registered [ 0.420143] acpiphp: Slot [25] registered [ 0.422142] acpiphp: Slot [26] registered [ 0.423109] acpiphp: Slot [27] registered [ 0.424147] acpiphp: Slot [28] registered [ 0.426190] acpiphp: Slot [29] registered [ 0.428145] acpiphp: Slot [30] registered [ 0.430139] acpiphp: Slot [31] registered [ 0.431088] PCI host bridge to bus 0000:00 [ 0.433040] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.436041] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.439056] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.443043] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.446024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.449026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.451193] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.455000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.458455] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.469745] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.475060] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.478018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.480018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.483022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.486359] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.489865] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.494055] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.497844] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.504017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.518018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.523932] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.529528] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.539018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.547016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.570017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.583567] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.595022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.604018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.622023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.633112] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.648016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.662020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.682027] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.695117] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.705016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.717018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.762019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.790837] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.797122] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.804018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.819023] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.830162] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.838018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.847019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.859019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.870959] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.876446] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.878427] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.881396] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.885305] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.890145] iommu: Default domain type: Passthrough [ 0.893429] SCSI subsystem initialized [ 0.894182] ACPI: bus type USB registered [ 0.896105] usbcore: registered new interface driver usbfs [ 0.897111] usbcore: registered new interface driver hub [ 0.899125] usbcore: registered new device driver usb [ 0.902314] pps_core: LinuxPPS API ver. 1 registered [ 0.904015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.907089] PTP clock support registered [ 0.910138] EDAC MC: Ver: 3.0.0 [ 0.912126] PCI: Using ACPI for IRQ routing [ 0.914070] NetLabel: Initializing [ 0.915012] NetLabel: domain hash size = 128 [ 0.916014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.917229] NetLabel: unlabeled traffic allowed by default [ 0.919190] vgaarb: loaded [ 0.921309] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.923025] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.930000] clocksource: Switched to clocksource kvm-clock [ 1.038745] VFS: Disk quotas dquot_6.6.0 [ 1.041513] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.044122] *** VALIDATE ramfs *** [ 1.045587] *** VALIDATE hugetlbfs *** [ 1.047196] pnp: PnP ACPI init [ 1.049375] pnp: PnP ACPI: found 6 devices [ 1.068301] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.073391] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.076034] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.078562] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.081356] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.083792] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.086554] NET: Registered protocol family 2 [ 1.089041] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.094052] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.098148] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.103855] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.108370] TCP: Hash tables configured (established 65536 bind 65536) [ 1.111582] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.115236] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.119794] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.122355] NET: Registered protocol family 1 [ 1.127291] RPC: Registered named UNIX socket transport module. [ 1.130001] RPC: Registered udp transport module. [ 1.131941] RPC: Registered tcp transport module. [ 1.133873] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.136533] NET: Registered protocol family 44 [ 1.138258] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.140744] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.143491] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.145740] PCI: CLS 0 bytes, default 64 [ 1.147621] Unpacking initramfs... [ 2.610392] debug: unmapping init [mem 0xffff92f1fcc54000-0xffff92f1fffbffff] [ 2.614689] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.617133] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.620306] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.114181] Initialise system trusted keyrings [ 3.116165] Key type blacklist registered [ 3.118715] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.128423] zbud: loaded [ 3.131857] *** VALIDATE nfs *** [ 3.133017] *** VALIDATE nfs4 *** [ 3.134331] pstore: using deflate compression [ 3.138439] Platform Keyring initialized [ 3.273464] NET: Registered protocol family 38 [ 3.275148] Key type asymmetric registered [ 3.276795] Asymmetric key parser 'x509' registered [ 3.278811] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.282024] io scheduler mq-deadline registered [ 3.283177] io scheduler kyber registered [ 3.284816] io scheduler bfq registered [ 3.286954] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.290098] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.293124] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.295567] ACPI: Power Button [PWRF] [ 3.300789] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.306417] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.317783] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.324175] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.341968] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.370303] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.398155] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.403867] Non-volatile memory driver v1.3 [ 3.406079] Linux agpgart interface v0.103 [ 3.440259] virtio_blk virtio1: [vda] 146864 512-byte logical blocks (75.2 MB/71.7 MiB) [ 3.445136] vda: detected capacity change from 0 to 75194368 [ 3.459709] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.462819] vdb: detected capacity change from 0 to 1073741824 [ 3.477636] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.481049] vdc: detected capacity change from 0 to 2621440000 [ 3.495481] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.498401] vdd: detected capacity change from 0 to 2621440000 [ 3.514164] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.516691] vde: detected capacity change from 0 to 4294967296 [ 3.534077] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.537782] vdf: detected capacity change from 0 to 4294967296 [ 3.559249] libphy: Fixed MDIO Bus: probed [ 3.567845] usbcore: registered new interface driver usbserial_generic [ 3.571025] usbserial: USB Serial support registered for generic [ 3.573801] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.579292] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.581561] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.584562] mousedev: PS/2 mouse device common for all mice [ 3.587827] rtc_cmos 00:05: RTC can wake from S4 [ 3.590994] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.594572] rtc_cmos 00:05: registered as rtc0 [ 3.598471] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.599036] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.601183] intel_pstate: CPU model not supported [ 3.608178] hid: raw HID events driver (C) Jiri Kosina [ 3.609927] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.610604] usbcore: registered new interface driver usbhid [ 3.617963] usbhid: USB HID core driver [ 3.619801] drop_monitor: Initializing network drop monitor service [ 3.622125] Initializing XFRM netlink socket [ 3.624323] NET: Registered protocol family 10 [ 3.627748] Segment Routing with IPv6 [ 3.629288] NET: Registered protocol family 17 [ 3.631887] mpls_gso: MPLS GSO support [ 3.637393] RAS: Correctable Errors collector initialized. [ 3.639778] AVX version of gcm_enc/dec engaged. [ 3.641598] AES CTR mode by8 optimization enabled [ 3.721721] sched_clock: Marking stable (3721579477, 0)->(4625948850, -904369373) [ 3.725700] registered taskstats version 1 [ 3.728159] Loading compiled-in X.509 certificates [ 3.730321] zswap: loaded using pool lzo/zbud [ 3.752821] Key type big_key registered [ 3.765157] Key type encrypted registered [ 3.767023] ima: No TPM chip found, activating TPM-bypass! [ 3.769205] ima: Allocated hash algorithm: sha1 [ 3.770904] ima: No architecture policies found [ 3.772813] evm: Initialising EVM extended attributes: [ 3.775067] evm: security.selinux [ 3.776449] evm: security.ima [ 3.777679] evm: security.capability [ 3.779089] evm: HMAC attrs: 0x1 [ 3.781717] rtc_cmos 00:05: setting system clock to 2026-08-27 20:19:10 UTC (1787861950) [ 3.788342] debug: unmapping init [mem 0xffffffff99603000-0xffffffff997fffff] [ 3.792080] debug: unmapping init [mem 0xffffffff98382000-0xffffffff98658fff] [ 3.801119] Write protecting the kernel read-only data: 28672k [ 3.804670] debug: unmapping init [mem 0xffffffff96a03000-0xffffffff96bfffff] [ 3.807967] debug: unmapping init [mem 0xffffffff97314000-0xffffffff973fffff] [ 3.850169] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.859942] systemd[1]: Detected virtualization kvm. [ 3.861705] systemd[1]: Detected architecture x86-64. [ 3.863193] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.888921] systemd[1]: No hostname configured. [ 3.890872] systemd[1]: Set hostname to . [ 3.892940] random: systemd: uninitialized urandom read (16 bytes read) [ 3.897863] systemd[1]: Initializing machine ID from random generator. [ 3.952514] random: ln: uninitialized urandom read (6 bytes read) [ 4.041285] random: systemd: uninitialized urandom read (16 bytes read) [ 4.044177] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.047961] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.053373] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.726663] device-mapper: uevent: version 1.0.3 [ 4.729122] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.495536] random: fast init done [ 5.533672] virtio_net virtio0 ens2: renamed from eth0 [ 5.541933] scsi host0: ata_piix [ 5.562282] scsi host1: ata_piix [ 5.563777] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.565858] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.317831] dracut-initqueue[589]: RTNETLINK answers: File exists [ 10.289612] random: crng init done [ 10.290872] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.759315] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.950732] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.199473] SELinux: Disabled at runtime. [ 12.261467] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.271794] systemd[1]: Detected virtualization kvm. [ 12.274071] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.741791] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.744628] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.750136] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.752689] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.754957] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.761067] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.764137] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Paths. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. [ 12.913418] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.136110] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.418375] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.523086] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.578332] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.586425] EDAC sbridge: Ver: 1.1.2 [ 15.102889] Key type dns_resolver registered [ 15.394238] NFS: Registering the id_resolver key type [ 15.396324] Key type id_resolver registered [ 15.398026] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Daily Cleanup of Temporary Directories. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg309-server login: [ 50.125145] hrtimer: interrupt took 9195872 ns [ 67.957555] libcfs: loading out-of-tree module taints kernel. [ 67.987152] Key type ._llcrypt registered [ 67.991692] Key type .llcrypt registered [ 68.072330] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 89.171152] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 90.859609] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 90.874704] alg: No test for adler32 (adler32-zlib) [ 92.317614] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 93.344165] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 95.079201] Key type lgssc registered [ 96.706599] Lustre: Echo OBD driver; http://www.lustre.org/ [ 114.728124] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 151.505783] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 162.793741] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 162.822142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 164.035730] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 164.080384] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 164.189939] Lustre: lustre-MDT0000: new disk, initializing [ 164.329554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 164.356105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 168.804270] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 182.876802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 182.995894] Lustre: 6521:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 183.037415] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 183.056478] Lustre: Skipped 1 previous similar message [ 183.113540] Lustre: lustre-MDT0001: new disk, initializing [ 183.171617] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 183.197826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 183.204954] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 186.881964] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 191.025499] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 201.485797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 201.704642] Lustre: lustre-OST0000: new disk, initializing [ 201.708413] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 201.713206] Lustre: 8455:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 201.785902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 207.938658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 207.948367] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 208.060592] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 208.866358] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 222.241524] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 222.380158] Lustre: lustre-OST0001: new disk, initializing [ 222.388292] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 222.399763] Lustre: 9529:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 222.507817] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 228.593909] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 231.007031] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 231.018107] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 231.075566] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 239.360867] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 245.538223] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 251.955421] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing check_logdir /tmp/testlogs/ [ 256.620441] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing yml_node [ 260.562729] Lustre: DEBUG MARKER: Client: 2.17.57.87 [ 262.773805] Lustre: DEBUG MARKER: MDS: 2.17.57.87 [ 264.900482] Lustre: DEBUG MARKER: OSS: 2.17.57.87 [ 266.305330] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 16:23:31 EDT 2026 [ 281.880386] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 283.341470] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 301.024360] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 301.041152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 301.069963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 303.082516] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 303.093552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 303.111283] Lustre: Skipped 2 previous similar messages [ 303.133460] Lustre: Skipped 3 previous similar messages [ 304.790629] Lustre: server umount lustre-MDT0000 complete [ 308.825658] LustreError: 6512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862255 with bad export cookie 16509274394761981330 [ 308.827809] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 308.845963] LustreError: 6512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 313.313079] LustreError: 6526:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 313.320193] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 313.327236] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 313.347123] LustreError: 6526:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 6 previous similar messages [ 313.355375] Lustre: Skipped 1 previous similar message [ 315.597606] Lustre: server umount lustre-MDT0001 complete [ 320.145322] Lustre: server umount lustre-OST0000 complete [ 323.817594] Lustre: server umount lustre-OST0001 complete [ 330.216023] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 339.942129] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 380.612887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 381.055101] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 381.087467] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 381.149730] Lustre: lustre-MDT0000: new disk, initializing [ 381.306586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 381.353496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 385.121906] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 394.284656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 394.368736] Lustre: 15082:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 394.431333] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 394.433694] Lustre: Skipped 1 previous similar message [ 394.476810] Lustre: lustre-MDT0001: new disk, initializing [ 394.681410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 394.739398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 394.754966] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 400.128579] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 410.348581] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 417.361686] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 423.521621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 423.865486] Lustre: lustre-OST0000: new disk, initializing [ 423.870659] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 423.878658] Lustre: 17050:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 424.041926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 425.866119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 425.879339] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 426.041933] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 430.292943] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 440.695252] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 445.026220] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 445.201554] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 449.582759] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 449.799470] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 456.678131] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 456.681333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 456.681563] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 456.691151] Lustre: Skipped 1 previous similar message [ 456.708133] Lustre: Skipped 2 previous similar messages [ 457.963374] Lustre: server umount lustre-OST0000 complete [ 461.801054] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 461.829150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 466.402772] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 466.418911] LustreError: Skipped 1 previous similar message [ 467.658345] Lustre: server umount lustre-MDT0000 complete [ 471.054058] LustreError: 15073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862417 with bad export cookie 16509274394761982870 [ 471.059245] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 471.070151] LustreError: 15073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 471.384641] Lustre: server umount lustre-MDT0001 complete [ 477.219693] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 16:27:02 (1787862422) [ 494.804318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 495.304821] LustreError: 19923:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 495.366482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 499.463847] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 500.707370] LustreError: 19924:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 505.826799] LustreError: 19923:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 507.363817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 512.387260] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 520.748839] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 526.286992] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 528.773045] Lustre: Failing over lustre-MDT0000 [ 529.099223] Lustre: server umount lustre-MDT0000 complete [ 531.425776] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 531.427749] LustreError: 19923:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 531.440413] Lustre: Skipped 2 previous similar messages [ 531.462079] LustreError: 19923:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 542.688386] LustreError: 19923:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 542.707101] LustreError: 19923:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 5 previous similar messages [ 547.627048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 547.763992] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 552.354869] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 553.187181] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 553.206976] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 553.222983] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 559.924672] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 563.681315] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 563.684790] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 563.690486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 563.693890] Lustre: Skipped 2 previous similar messages [ 569.665858] Lustre: server umount lustre-MDT0000 complete [ 572.896591] LustreError: 19918:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 572.918526] LustreError: 19918:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 573.452607] LustreError: 19905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862520 with bad export cookie 16509274394761984284 [ 573.461575] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 573.734522] Lustre: server umount lustre-MDT0001 complete [ 582.995378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 583.599244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 583.608451] Lustre: Skipped 2 previous similar messages [ 588.571779] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 597.658927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 601.719529] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 608.499402] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 614.258878] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 616.277687] Lustre: Failing over lustre-MDT0000 [ 616.287860] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 616.295828] Lustre: Skipped 2 previous similar messages [ 616.296237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 616.305505] Lustre: Skipped 3 previous similar messages [ 616.477753] Lustre: server umount lustre-MDT0000 complete [ 621.536601] LustreError: 23252:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 621.551547] LustreError: 23252:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 634.507442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 634.617677] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 634.876564] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 638.529387] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 639.974374] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 639.979317] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 639.992866] Lustre: Skipped 1 previous similar message [ 640.015661] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 647.250852] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 651.054799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 651.068739] Lustre: Skipped 1 previous similar message [ 657.463805] Lustre: server umount lustre-MDT0000 complete [ 661.325274] LustreError: 23236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862608 with bad export cookie 16509274394761985222 [ 661.336228] LustreError: 23236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 661.342600] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 661.788942] Lustre: server umount lustre-MDT0001 complete [ 669.678200] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 16:30:14 (1787862614) [ 685.334231] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 695.424342] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 732.839755] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 743.554056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 743.783125] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 743.815846] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 743.931124] Lustre: lustre-MDT0000: new disk, initializing [ 744.027698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 744.034180] Lustre: Skipped 2 previous similar messages [ 744.047590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 748.117863] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 757.946597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 758.061420] Lustre: 29635:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 758.323206] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 762.440831] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 771.197735] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 775.752545] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 781.761536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 782.117181] Lustre: 31602:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 783.996962] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 784.047847] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 784.127498] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 784.147636] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 788.186688] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 796.368134] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 800.822556] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 820.193690] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 820.199039] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 820.201580] LustreError: Skipped 1 previous similar message [ 820.204146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 820.204155] Lustre: Skipped 2 previous similar messages [ 820.232974] Lustre: Skipped 3 previous similar messages [ 824.995877] Lustre: server umount lustre-OST0000 complete [ 829.924364] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 834.658747] Lustre: server umount lustre-MDT0000 complete [ 835.555615] LustreError: 29646:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 835.578510] LustreError: 29646:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 7 previous similar messages [ 838.083984] LustreError: 29628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862784 with bad export cookie 16509274394761985628 [ 838.085323] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 838.094158] LustreError: 29628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 838.478517] Lustre: server umount lustre-MDT0001 complete [ 846.804081] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 849.168727] Key type lgssc unregistered [ 849.459812] LNet: 33581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 849.474853] LNetError: 33581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 849.490528] LNet: Removed LNI 192.168.203.109@tcp [ 850.395279] Key type .llcrypt unregistered [ 850.398996] Key type ._llcrypt unregistered [ 875.865693] Key type ._llcrypt registered [ 875.867697] Key type .llcrypt registered [ 875.943652] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 876.713851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 876.800372] alg: No test for adler32 (adler32-zlib) [ 877.805969] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 878.038291] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 879.735233] Key type lgssc registered [ 880.745950] Lustre: Echo OBD driver; http://www.lustre.org/ [ 887.876795] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 887.895968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 889.454718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 893.481210] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 900.068740] LustreError: 34995:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 900.093544] LustreError: 34995:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 900.506495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 900.876861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 904.935474] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 911.640814] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 916.893595] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 923.616454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 923.982223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 926.008866] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 929.540547] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 935.244544] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 936.177584] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 949.208630] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 16:34:54 (1787862894) [ 950.764370] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 952.390803] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 958.086477] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 16:35:03 (1787862903) [ 965.259592] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 16:35:10 (1787862910) [ 966.317231] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 974.622365] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 974.632367] Lustre: Skipped 2 previous similar messages [ 976.230771] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 16:35:21 (1787862921) [ 979.618134] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 979.623346] Lustre: Skipped 3 previous similar messages [ 995.080444] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 995.085161] Lustre: Skipped 6 previous similar messages [ 997.051338] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 16:35:42 (1787862942) [ 999.431594] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1002.567928] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1009.417276] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 16:35:54 (1787862954) [ 1011.760493] Lustre: Disabling parameter general.jobid_name= in log params [ 1011.764518] Lustre: Skipped 4 previous similar messages [ 1021.789524] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 16:36:07 (1787862967) [ 1037.798593] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1037.799705] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1037.805563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1037.812204] Lustre: Skipped 1 previous similar message [ 1040.671597] Lustre: server umount lustre-OST0000 complete [ 1044.454098] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1044.459105] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1044.475720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1044.482782] Lustre: Skipped 1 previous similar message [ 1048.033907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1048.036060] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1053.153119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1053.166205] Lustre: Skipped 2 previous similar messages [ 1058.271424] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1058.275162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1058.296367] Lustre: Skipped 1 previous similar message [ 1058.526477] Lustre: server umount lustre-MDT0000 complete [ 1061.718747] LustreError: 34977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863008 with bad export cookie 14968562828189457065 [ 1061.725056] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1061.729582] LustreError: 34977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1062.132037] Lustre: server umount lustre-MDT0001 complete [ 1072.355804] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 1075.329185] Key type lgssc unregistered [ 1075.640105] LNet: 42855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1075.647691] LNetError: 42855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1075.668431] LNet: Removed LNI 192.168.203.109@tcp [ 1076.608527] Key type .llcrypt unregistered [ 1076.615283] Key type ._llcrypt unregistered [ 1093.656601] Key type ._llcrypt registered [ 1093.658598] Key type .llcrypt registered [ 1093.746577] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 1108.346778] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 1109.311608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1109.376863] alg: No test for adler32 (adler32-zlib) [ 1110.413970] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 1110.650818] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 1112.343317] Key type lgssc registered [ 1113.185770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1141.701252] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1141.737822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1143.074476] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1143.118320] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1143.197658] Lustre: lustre-MDT0000: new disk, initializing [ 1143.282686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1143.306908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1148.554553] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1160.246235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1160.446401] Lustre: 46007:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1160.579377] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1160.584476] Lustre: Skipped 1 previous similar message [ 1160.659816] Lustre: lustre-MDT0001: new disk, initializing [ 1160.730168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1160.756519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1160.765263] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1166.830482] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1176.657500] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1181.927426] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1189.160703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1189.559120] Lustre: lustre-OST0000: new disk, initializing [ 1189.562166] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1189.575159] Lustre: 47979:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1189.712569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1196.128348] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1197.087379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1197.106253] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1197.237625] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1204.644409] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1210.237639] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1210.532210] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1214.997227] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1215.173597] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1217.504443] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1217.519497] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1217.532589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1223.138792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1223.143261] Lustre: Skipped 3 previous similar messages [ 1228.262249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1228.270804] Lustre: Skipped 1 previous similar message [ 1231.327252] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1231.497599] Lustre: server umount lustre-OST0000 complete [ 1237.473642] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1237.484969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1237.502612] Lustre: Skipped 1 previous similar message [ 1237.512404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1238.499786] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1241.325190] Lustre: server umount lustre-MDT0000 complete [ 1244.760093] LustreError: 45999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863191 with bad export cookie 7895265164765896805 [ 1244.773318] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1244.774968] LustreError: 45999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1245.098279] Lustre: server umount lustre-MDT0001 complete [ 1254.065717] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 16:39:58 (1787863198) [ 1260.205415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1260.764973] LustreError: 50438:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1260.785639] LustreError: 50438:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 1260.906391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1265.385526] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1266.144450] LustreError: 50439:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1271.269569] LustreError: 50438:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1272.983043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1273.419104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1277.499210] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1284.731509] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1289.834441] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1296.390697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1296.731128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1301.423944] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1307.829488] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1313.590881] LustreError: 52966:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1314.656348] Lustre: Modifying parameter general.timeout=1 in log params [ 1316.098941] Lustre: Modifying parameter general.timeout=3 in log params [ 1316.105718] Lustre: Skipped 1 previous similar message [ 1318.695538] Lustre: Modifying parameter general.timeout=6 in log params [ 1318.704510] Lustre: Skipped 2 previous similar messages [ 1322.825438] Lustre: Modifying parameter general.timeout=10 in log params [ 1322.834118] Lustre: Skipped 3 previous similar messages [ 1331.363384] Lustre: Modifying parameter general.timeout=19 in log params [ 1331.371667] Lustre: Skipped 8 previous similar messages [ 1348.132486] Lustre: Modifying parameter general.timeout=39 in log params [ 1348.139312] Lustre: Skipped 19 previous similar messages [ 1380.842449] Lustre: Modifying parameter general.timeout=73 in log params [ 1380.850801] Lustre: Skipped 33 previous similar messages [ 1437.472215] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 16:43:02 (1787863382) [ 1440.539269] Lustre: 59913:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1441.974456] Lustre: 60011:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1444.802163] Lustre: 60225:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1444.827105] Lustre: 60225:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1445.665557] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 in log params [ 1445.683866] Lustre: Skipped 62 previous similar messages [ 1475.726889] Lustre: 61787:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1475.739962] Lustre: 61787:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1475.782290] LustreError: 61787:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1479.500816] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 16:43:44 (1787863424) [ 1492.961718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1492.965339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1492.979819] Lustre: Skipped 2 previous similar messages [ 1492.999044] Lustre: Skipped 4 previous similar messages [ 1498.083697] LustreError: 50433:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1498.094904] LustreError: 50433:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1498.212504] Lustre: server umount lustre-MDT0000 complete [ 1501.756885] LustreError: 54209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863448 with bad export cookie 7895265164765897687 [ 1501.763027] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1501.772554] LustreError: 54209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1501.977548] Lustre: server umount lustre-MDT0001 complete [ 1515.964839] Lustre: server umount lustre-OST0000 complete [ 1547.210729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1547.570202] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1547.882443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1552.111714] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1560.891247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1560.989265] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1561.016342] Lustre: 64320:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1566.205567] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1573.903164] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1579.413198] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1586.183095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1586.347031] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1586.613285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1586.617281] Lustre: Skipped 1 previous similar message [ 1592.101398] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1599.660884] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1601.595701] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1601.609803] Lustre: Skipped 17 previous similar messages [ 1601.660178] Lustre: 66886:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1611.237907] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1611.252533] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1611.269300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1615.668379] Lustre: server umount lustre-OST0000 complete [ 1621.994188] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1622.019165] Lustre: Skipped 2 previous similar messages [ 1632.225250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1632.234252] Lustre: Skipped 6 previous similar messages [ 1634.271398] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1634.638605] Lustre: server umount lustre-MDT0000 complete [ 1637.344236] LustreError: 64332:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1637.389582] LustreError: 64332:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 1639.050284] LustreError: 65061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863585 with bad export cookie 7895265164765900704 [ 1639.054757] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1639.082708] LustreError: 65061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1639.483117] Lustre: server umount lustre-MDT0001 complete [ 1650.882893] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 1653.777808] Key type lgssc unregistered [ 1654.110828] LNet: 68100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1654.116679] LNetError: 68100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1654.129909] LNet: Removed LNI 192.168.203.109@tcp [ 1655.064760] Key type .llcrypt unregistered [ 1655.070089] Key type ._llcrypt unregistered [ 1682.345934] Key type ._llcrypt registered [ 1682.352859] Key type .llcrypt registered [ 1682.632471] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 1684.423446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1684.451305] alg: No test for adler32 (adler32-zlib) [ 1685.581737] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 1685.975076] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 1687.711493] Key type lgssc registered [ 1689.433991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1697.697977] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1697.747834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1699.131471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1703.633722] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1708.526542] LustreError: 69513:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1708.540328] LustreError: 69513:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1711.076565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1712.287032] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1717.656911] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1725.816329] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1730.706418] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1737.832526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1738.371903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1744.504606] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1751.756872] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1763.593736] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1772.518695] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1772.522453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1772.530655] Lustre: Skipped 1 previous similar message [ 1772.552617] Lustre: Skipped 2 previous similar messages [ 1777.638500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1777.658915] Lustre: Skipped 2 previous similar messages [ 1778.308558] Lustre: server umount lustre-MDT0000 complete [ 1782.100075] LustreError: 69493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863728 with bad export cookie 8793203070393407090 [ 1782.110392] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1782.116255] LustreError: 69493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1782.498174] Lustre: server umount lustre-MDT0001 complete [ 1795.690775] Lustre: server umount lustre-OST0000 complete [ 1821.149154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1821.500387] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1821.529128] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1821.780924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1825.469831] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1834.072169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1834.224345] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1834.266310] Lustre: 74387:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1834.787167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1839.662636] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1846.717839] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1852.153462] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1858.846635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1859.001741] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1859.272082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1865.311457] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1872.696628] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1874.289649] Lustre: 76959:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1875.931627] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1882.080506] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1882.085558] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1882.094260] Lustre: Skipped 1 previous similar message [ 1882.099917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1882.603693] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1885.817861] Lustre: server umount lustre-OST0000 complete [ 1890.795318] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1890.816186] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1890.847874] Lustre: Skipped 1 previous similar message [ 1890.863705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1890.870492] Lustre: Skipped 1 previous similar message [ 1892.839422] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1895.814892] Lustre: server umount lustre-MDT0000 complete [ 1897.952436] LustreError: 74392:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1897.984788] LustreError: 74392:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 1899.272106] LustreError: 74378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863845 with bad export cookie 8793203070393407923 [ 1899.275125] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1899.286292] LustreError: 74378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1899.866395] Lustre: server umount lustre-MDT0001 complete [ 1910.496859] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 1913.394565] Key type lgssc unregistered [ 1913.688582] LNet: 78172:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1913.703947] LNetError: 78172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1913.718882] LNet: Removed LNI 192.168.203.109@tcp [ 1914.547618] Key type .llcrypt unregistered [ 1914.549889] Key type ._llcrypt unregistered [ 1941.340967] Key type ._llcrypt registered [ 1941.343157] Key type .llcrypt registered [ 1941.440681] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 1942.547090] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1942.585574] alg: No test for adler32 (adler32-zlib) [ 1943.696588] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 1943.977641] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 1945.711239] Key type lgssc registered [ 1946.908399] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1954.617747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1954.636155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1956.062106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1959.895846] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1965.537131] LustreError: 79586:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1965.553203] LustreError: 79586:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1967.477794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1968.056346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1971.142707] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1976.964210] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1982.821433] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1990.691599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1991.470877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1998.452968] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2005.440968] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2010.617766] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2011.328734] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2012.528163] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2012.534637] Lustre: Skipped 1 previous similar message [ 2015.037989] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2015.041579] Lustre: Skipped 2 previous similar messages [ 2019.259377] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2019.268581] Lustre: Skipped 4 previous similar messages [ 2027.838526] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2027.843488] Lustre: Skipped 9 previous similar messages [ 2044.649745] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2044.654361] Lustre: Skipped 18 previous similar messages [ 2077.414426] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2077.427744] Lustre: Skipped 33 previous similar messages [ 2109.396844] Lustre: 87689:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2111.872518] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 16:54:16 (1787864056) [ 2113.619822] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2115.494789] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 16:54:20 (1787864060) [ 2120.678743] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2120.693968] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2120.716021] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2123.751837] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2123.755209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2123.771223] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2123.778424] Lustre: Skipped 1 previous similar message [ 2126.034236] Lustre: server umount lustre-OST0000 complete [ 2131.936390] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2131.941977] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2131.950846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2133.985462] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2133.988651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2134.022945] Lustre: Skipped 1 previous similar message [ 2135.560282] Lustre: server umount lustre-MDT0000 complete [ 2139.007454] LustreError: 79567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864085 with bad export cookie 428344026007056389 [ 2139.009373] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2139.019191] LustreError: 79567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2139.308618] Lustre: server umount lustre-MDT0001 complete [ 2149.262725] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 2152.223557] Key type lgssc unregistered [ 2152.498584] LNet: 88940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2152.511442] LNetError: 88940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2152.541877] LNet: Removed LNI 192.168.203.109@tcp [ 2153.371368] Key type .llcrypt unregistered [ 2153.376747] Key type ._llcrypt unregistered [ 2157.710450] Key type ._llcrypt registered [ 2157.712732] Key type .llcrypt registered [ 2157.801364] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2162.592145] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules [ 2164.036048] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2164.111161] alg: No test for adler32 (adler32-zlib) [ 2165.241228] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 2165.638713] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 2167.362151] Key type lgssc registered [ 2168.683851] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2169.955531] LustreError: 90163:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2185.302412] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 2190.940331] LustreError: 90163:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2193.943731] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2193.962447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2195.300579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2199.759527] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2204.648471] LustreError: 91167:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2204.664311] LustreError: 91167:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2208.099363] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 16:55:53 (1787864153) [ 2209.762677] LustreError: 91166:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2212.218561] LustreError: 91198:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2212.224801] LustreError: 91198:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2212.232254] LustreError: 91198:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 0, failed: rc = -5 [ 2212.533963] Lustre: server umount lustre-MDT0000 complete [ 2226.194526] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 2228.750935] Key type lgssc unregistered [ 2229.026450] LNet: 92694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2229.035398] LNetError: 92694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2229.055925] LNet: Removed LNI 192.168.203.109@tcp [ 2229.926636] Key type .llcrypt unregistered [ 2229.930404] Key type ._llcrypt unregistered [ 2246.995622] Key type ._llcrypt registered [ 2246.998557] Key type .llcrypt registered [ 2247.085512] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 2247.780134] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2247.798212] alg: No test for adler32 (adler32-zlib) [ 2248.878566] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 2249.072537] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 2250.727191] Key type lgssc registered [ 2251.636315] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2258.248236] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2258.272930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2259.917254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2264.456649] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2270.181323] LustreError: 93910:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2270.194204] LustreError: 93910:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2272.499142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2273.210824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2277.834777] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2285.285584] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2290.735634] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2296.877713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2297.637833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2302.877449] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2310.338716] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2357.439643] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 16:58:22 (1787864302) [ 2359.809218] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2361.657404] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 16:58:26 (1787864306) [ 2370.021783] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2370.031544] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2370.049859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2371.553525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2371.560390] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2371.571951] Lustre: Skipped 1 previous similar message [ 2371.580136] Lustre: Skipped 1 previous similar message [ 2373.016779] Lustre: server umount lustre-MDT0000 complete [ 2376.484306] LustreError: 93891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864323 with bad export cookie 17482423541135810888 [ 2376.486191] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2376.489031] LustreError: 93891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2376.659650] Lustre: server umount lustre-MDT0001 complete [ 2390.559271] Lustre: server umount lustre-OST0000 complete [ 2401.902259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2402.667552] LustreError: 98187:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2402.941486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2407.765202] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2408.427437] LustreError: 98188:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2412.533597] LustreError: 98187:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2416.784831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2417.434484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2422.776940] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2429.726231] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2433.625334] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2444.244245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2444.484162] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2444.490191] LustreError: 98177:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2444.511745] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2444.526763] LustreError: 100270:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2444.541430] LustreError: 100270:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2444.546511] LustreError: 100270:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2444.663140] Lustre: server umount lustre-OST0000 complete [ 2444.665551] LustreError: 100270:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2452.424414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2452.606186] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2452.618354] LustreError: 98176:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2452.635184] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2452.644153] LustreError: 100731:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2452.651434] LustreError: 100731:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2452.658431] LustreError: 100731:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2452.721929] Lustre: server umount lustre-OST0000 complete [ 2452.724711] LustreError: 100731:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2459.973289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2460.237926] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2460.245757] Lustre: Found index 0 for lustre-OST0000, updating log [ 2460.258950] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2460.385914] Lustre: lustre-OST0000: new disk, initializing [ 2460.391681] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2460.398085] Lustre: 101267:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2460.641098] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2467.185115] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2470.792644] Lustre: Failing over lustre-OST0000 [ 2470.919768] Lustre: server umount lustre-OST0000 complete [ 2478.562032] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2478.572740] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2478.586086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2480.959329] Lustre: server umount lustre-MDT0000 complete [ 2484.193917] LustreError: 98188:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2484.207976] LustreError: 98188:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2484.596285] LustreError: 101266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864431 with bad export cookie 17482423541135813625 [ 2484.599936] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2484.605022] LustreError: 101266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2495.144536] Lustre: server umount lustre-MDT0001 complete [ 2503.065211] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 17:00:48 (1787864448) [ 2520.109511] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 2530.430206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2530.702697] LustreError: 103626:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2530.806833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2534.645814] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2542.313709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2548.255322] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2551.172916] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2557.523219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2558.233450] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2563.168593] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2567.462395] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2567.483174] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2567.485670] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2568.625459] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2568.662323] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2568.711406] Lustre: lustre-OST0001: new disk, initializing [ 2568.714569] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2568.718604] Lustre: 105946:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2568.974674] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2568.979642] Lustre: Skipped 2 previous similar messages [ 2574.290322] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2576.505141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2576.518604] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2576.616600] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2581.296421] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2586.897702] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2592.234921] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2592.239620] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2592.277638] Lustre: Skipped 3 previous similar messages [ 2592.282236] Lustre: Skipped 3 previous similar messages [ 2597.152025] Lustre: server umount lustre-MDT0001 complete [ 2597.351694] LustreError: 104358:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2597.360395] LustreError: 104358:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 2606.211290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2606.315630] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2606.320103] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2606.364302] Lustre: 103614:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2607.750533] LustreError: 107503:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2607.757233] LustreError: 107503:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2607.761283] LustreError: 103618:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2607.869835] LustreError: 107520:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.109@tcp: cfg command failed: rc = -17 [ 2607.885282] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2607.885282] [ 2610.882852] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2612.215098] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2619.834157] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 17:02:44 (1787864564) [ 2636.633480] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 2656.530706] Lustre: 109561:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2676.320308] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2679.981875] Lustre: 110697:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2755.041714] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2755.049805] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2755.070620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2755.075852] Lustre: Skipped 2 previous similar messages [ 2760.986684] Lustre: server umount lustre-MDT0000 complete [ 2764.604501] LustreError: 103607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864711 with bad export cookie 17482423541135814563 [ 2764.613129] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2764.616091] LustreError: 103607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2765.020239] Lustre: server umount lustre-MDT0001 complete [ 2779.467375] Lustre: server umount lustre-OST0000 complete [ 2782.303088] Lustre: 93222:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787864712/real 1787864712] req@ffff92f14be25e00 x1874711486480256/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787864728 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2782.331604] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2782.340943] Lustre: Skipped 3 previous similar messages [ 2783.093486] Lustre: server umount lustre-OST0001 complete [ 2788.973664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2798.276419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2805.906556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2814.723092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2831.936664] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 2840.637533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2840.654261] Lustre: lustre-MDT0000: reset Object Index mappings [ 2841.022494] LustreError: 114642:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2841.041958] LustreError: 114642:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 5 previous similar messages [ 2841.126360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2841.135481] Lustre: Skipped 1 previous similar message [ 2845.185392] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2854.031111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2854.050301] Lustre: lustre-MDT0001: reset Object Index mappings [ 2859.895348] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2863.313577] Lustre: 115791:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2869.617606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2877.385892] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2879.559983] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:231 to 0x280000400:257) [ 2885.696902] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2887.556010] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 2887.577759] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 2887.647596] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:157 to 0x2c0000401:193) [ 2892.708508] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2899.957628] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2903.250668] Lustre: 117668:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2951.745331] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 17:08:16 (1787864896) [ 2979.825852] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2979.843639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2979.858302] Lustre: Skipped 2 previous similar messages [ 2979.881688] Lustre: Skipped 10 previous similar messages [ 2984.382209] Lustre: server umount lustre-MDT0000 complete [ 2984.933719] LustreError: 114643:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2984.957333] LustreError: 114643:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 7 previous similar messages [ 2988.095948] LustreError: 114624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864934 with bad export cookie 17482423541135861456 [ 2988.111257] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2988.126496] LustreError: 114624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2994.652586] Lustre: server umount lustre-MDT0001 complete [ 3004.728120] Lustre: server umount lustre-OST0000 complete [ 3007.641240] LustreError: 118631:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0001_UUID: namespace resource [0x2c0000400:0x3f:0x0].0x0 (ffff92f2612ac700) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 3014.203406] Lustre: server umount lustre-OST0001 complete [ 3022.550684] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 3034.685835] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 3071.611703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3071.846756] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3071.887862] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3071.970781] Lustre: lustre-MDT0000: new disk, initializing [ 3072.034575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3072.041780] Lustre: Skipped 3 previous similar messages [ 3072.052992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3077.043650] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3084.850392] Lustre: server umount lustre-MDT0000 complete [ 3095.408281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3095.935083] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3095.966133] Lustre: 121943:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3095.996746] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3101.806780] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3107.621514] Lustre: server umount lustre-MDT0000 complete [ 3116.600809] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 17:11:01 (1787865061) [ 3118.136447] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3119.830962] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 17:11:05 (1787865065) [ 3138.010575] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 3149.700804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3154.247794] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3162.857506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3162.955994] Lustre: 123992:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3163.005411] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3163.009775] Lustre: Skipped 1 previous similar message [ 3163.080988] Lustre: lustre-MDT0001: new disk, initializing [ 3163.300491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3163.317955] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3168.435723] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3173.008464] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3179.992570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3180.402983] Lustre: lustre-OST0000: new disk, initializing [ 3180.407573] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3180.415816] Lustre: 125573:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3181.673048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3181.688638] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3181.845414] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3187.600994] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3198.489376] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3198.759269] Lustre: lustre-OST0001: new disk, initializing [ 3198.765277] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3198.779048] Lustre: 126443:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3200.734854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3200.751179] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3200.924867] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3206.440547] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3216.277803] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3220.367741] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3631.213683] Lustre: 132864:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3631.221956] Lustre: 132864:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3631.230230] Lustre: 132864:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3631.235676] Lustre: 132864:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3631.240944] Lustre: 132864:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3631.245624] Lustre: 132864:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3925.209386] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 17:24:29 (1787865869) [ 3941.346980] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3941.352232] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3941.367402] Lustre: Skipped 3 previous similar messages [ 3941.380204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3941.387824] Lustre: Skipped 4 previous similar messages [ 3943.398519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3943.403847] Lustre: Skipped 3 previous similar messages [ 3947.173237] Lustre: server umount lustre-MDT0000 complete [ 3948.527281] LustreError: 124000:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3948.552042] LustreError: 124000:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 7 previous similar messages [ 3950.580123] LustreError: 123985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865897 with bad export cookie 17482423541135928922 [ 3950.599443] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3950.601370] LustreError: 123985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3953.633991] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3953.666292] Lustre: Skipped 3 previous similar messages [ 3953.688860] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3953.704208] Lustre: Skipped 1 previous similar message [ 3957.138545] Lustre: server umount lustre-MDT0001 complete [ 3967.406869] Lustre: server umount lustre-OST0000 complete [ 3985.377965] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3985.561897] Lustre: server umount lustre-OST0001 complete [ 3992.351328] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 4003.015314] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 4037.555617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4037.964902] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4038.043746] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4038.187754] Lustre: lustre-MDT0000: new disk, initializing [ 4038.343098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4038.360572] Lustre: Skipped 5 previous similar messages [ 4038.408266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4042.689929] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4054.659642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4054.781349] Lustre: 135925:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4054.852216] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4054.855815] Lustre: Skipped 1 previous similar message [ 4054.974113] Lustre: lustre-MDT0001: new disk, initializing [ 4055.205041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4055.228149] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4060.164538] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4070.383774] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4076.379951] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4083.033021] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4083.321496] Lustre: lustre-OST0000: new disk, initializing [ 4083.326387] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4083.332451] Lustre: 137894:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4083.449744] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4083.460144] Lustre: Skipped 1 previous similar message [ 4085.389817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4085.404469] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4085.523590] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4089.296427] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4097.179143] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4105.833478] Lustre: lustre-MDD0000: changelog on [ 4108.230794] Lustre: lustre-MDD0001: changelog on [ 5853.401080] Lustre: lustre-MDD0001: changelog off [ 5855.335269] Lustre: 136809:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5856.308964] Lustre: lustre-MDD0000: changelog off [ 5867.493080] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5867.495203] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5867.500507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5867.500515] Lustre: Skipped 2 previous similar messages [ 5867.510894] Lustre: Skipped 2 previous similar messages [ 5869.316694] Lustre: server umount lustre-OST0000 complete [ 5877.734348] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5877.735402] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5877.741709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5877.741717] Lustre: Skipped 1 previous similar message [ 5877.756568] Lustre: Skipped 1 previous similar message [ 5879.441118] Lustre: server umount lustre-MDT0000 complete [ 5882.796645] LustreError: 136721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867829 with bad export cookie 17482423541136749798 [ 5882.801401] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5882.807795] LustreError: 136721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5883.153888] Lustre: server umount lustre-MDT0001 complete [ 5895.065484] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 5898.040108] Key type lgssc unregistered [ 5898.311667] LNet: 147016:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5898.329308] LNetError: 147016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5898.341697] LNet: Removed LNI 192.168.203.109@tcp [ 5899.417155] Key type .llcrypt unregistered [ 5899.420526] Key type ._llcrypt unregistered [ 5918.168603] Key type ._llcrypt registered [ 5918.171878] Key type .llcrypt registered [ 5918.278468] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 5935.087269] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 5936.754861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5936.790337] alg: No test for adler32 (adler32-zlib) [ 5937.939690] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 5938.205329] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 5939.888458] Key type lgssc registered [ 5941.108906] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5975.447472] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5975.513060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5976.694097] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5976.718368] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5976.814458] Lustre: lustre-MDT0000: new disk, initializing [ 5976.901697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5976.922960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5981.428859] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 5991.426793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5991.576786] Lustre: 150220:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5991.623304] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5991.626428] Lustre: Skipped 1 previous similar message [ 5991.709606] Lustre: lustre-MDT0001: new disk, initializing [ 5991.847634] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5991.903952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5991.914551] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5996.588942] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6006.323568] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6012.187231] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6018.390934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6018.727173] Lustre: lustre-OST0000: new disk, initializing [ 6018.737517] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6018.767511] Lustre: 152189:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6018.902583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6025.317518] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6027.355036] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6027.363749] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6027.501286] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6035.038666] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6042.009946] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_module obdecho/obdecho [ 6044.275562] LustreError: 153192:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6044.283252] LustreError: 153192:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6046.256426] LustreError: 153289:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6058.470981] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6058.476337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6058.481854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6058.491753] Lustre: Skipped 2 previous similar messages [ 6058.495783] Lustre: Skipped 1 previous similar message [ 6063.590056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6063.596123] Lustre: Skipped 2 previous similar messages [ 6064.010749] Lustre: server umount lustre-MDT0000 complete [ 6068.235540] LustreError: 150211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868014 with bad export cookie 5069906498968279857 [ 6068.237815] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6068.254275] LustreError: 150211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6068.754498] Lustre: server umount lustre-MDT0001 complete [ 6083.221621] Lustre: server umount lustre-OST0000 complete [ 6106.080646] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 6116.934901] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 6152.268564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6152.469352] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6152.497590] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6152.594705] Lustre: lustre-MDT0000: new disk, initializing [ 6152.670884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6152.687055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6156.628631] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6171.479614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6171.658164] Lustre: 157015:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6171.819291] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6171.823019] Lustre: Skipped 1 previous similar message [ 6171.939782] Lustre: lustre-MDT0001: new disk, initializing [ 6172.112410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6172.164914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6172.172633] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6178.576068] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6187.251569] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6192.278626] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6198.408459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6198.651234] Lustre: lustre-OST0000: new disk, initializing [ 6198.656339] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6198.668964] Lustre: 158989:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6198.764960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6204.573895] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6205.540771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6205.565348] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6205.681888] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6214.095402] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6218.884613] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6219.102805] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6224.527082] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6224.859332] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6231.523404] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6231.525737] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6231.531234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6231.538640] LustreError: Skipped 1 previous similar message [ 6231.545299] Lustre: Skipped 1 previous similar message [ 6232.904415] Lustre: server umount lustre-OST0000 complete [ 6238.690063] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6238.696039] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6238.703233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6238.707288] Lustre: Skipped 1 previous similar message [ 6241.767435] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6243.216360] Lustre: server umount lustre-MDT0000 complete [ 6247.318968] LustreError: 157009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868194 with bad export cookie 5069906498968280823 [ 6247.341848] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6247.349342] LustreError: 157009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6247.831802] Lustre: server umount lustre-MDT0001 complete [ 6252.024757] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 18:03:16 (1787868196) [ 6254.012175] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6256.043964] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 18:03:20 (1787868200) [ 6264.942058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6265.439215] LustreError: 161402:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6265.475856] LustreError: 161402:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 6265.583749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6270.947031] LustreError: 161403:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6270.953287] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6276.065220] LustreError: 161402:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6279.721796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6284.502652] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6291.868317] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6297.221330] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6304.162410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6304.514545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6304.521672] Lustre: Skipped 1 previous similar message [ 6310.643501] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6318.919878] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6326.240685] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6326.250303] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6326.265034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6326.271182] Lustre: Skipped 2 previous similar messages [ 6326.288124] Lustre: Skipped 4 previous similar messages [ 6331.315367] Lustre: server umount lustre-MDT0000 complete [ 6331.381108] LustreError: 161403:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6331.404450] LustreError: 161403:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 6336.484118] LustreError: 161402:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6336.513627] LustreError: 161402:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 6336.667580] LustreError: 161383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868283 with bad export cookie 5069906498968281705 [ 6336.673689] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6336.678925] LustreError: 161383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6340.257592] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.9@tcp (stopping) [ 6340.261594] Lustre: Skipped 1 previous similar message [ 6341.600289] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6343.069517] Lustre: server umount lustre-MDT0001 complete [ 6349.876450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6360.756686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6361.107366] LustreError: 165243:0:(llog.c:1655:llog_backup()) MGC192.168.203.109@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6361.138729] Lustre: 165243:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.109@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6362.082460] LustreError: 165248:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6362.123116] LustreError: 165248:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 6380.145121] LustreError: 165250:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6380.162681] LustreError: 165250:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 8 previous similar messages [ 6402.033717] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6402.046427] Lustre: Skipped 1 previous similar message [ 6416.866619] LustreError: 165248:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6416.889074] LustreError: 165248:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 28 previous similar messages [ 6422.495948] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6422.501514] Lustre: 165243:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.109@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6463.472514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6466.911553] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6468.598132] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6474.674263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6481.895638] LustreError: 165249:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6481.898428] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x465bed196b72fb9f [ 6481.903353] LustreError: 165249:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 45 previous similar messages [ 6481.926874] Lustre: MGC192.168.203.109@tcp: Connection restored to 0@lo (at 0@lo) [ 6485.816084] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6487.532085] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6493.516514] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6495.139042] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6500.181457] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6501.652180] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6518.242800] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6518.245124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6518.258721] Lustre: Skipped 1 previous similar message [ 6518.267859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6524.473353] Lustre: server umount lustre-MDT0000 complete [ 6527.882816] LustreError: 161383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868474 with bad export cookie 5069906498968288159 [ 6527.886839] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6527.895642] LustreError: 161383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6528.116272] Lustre: server umount lustre-MDT0001 complete [ 6541.858151] Lustre: server umount lustre-OST0000 complete [ 6562.846552] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 6572.190590] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 6608.967871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6609.146841] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6609.172844] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6609.261362] Lustre: lustre-MDT0000: new disk, initializing [ 6609.346316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6609.358970] Lustre: Skipped 1 previous similar message [ 6609.377648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6613.551308] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6623.288241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6623.413640] Lustre: 170348:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6623.680942] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6628.069589] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6637.095276] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6642.432205] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6649.028250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6649.234471] Lustre: lustre-OST0000: new disk, initializing [ 6649.237315] Lustre: Skipped 1 previous similar message [ 6649.240915] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6649.244904] Lustre: Skipped 2 previous similar messages [ 6649.249261] Lustre: 172314:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6654.522230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6654.532486] Lustre: Skipped 1 previous similar message [ 6654.539952] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6654.653757] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6655.259435] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6664.454863] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6669.040680] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6669.224522] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6673.898486] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6674.113287] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6680.554183] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6680.556564] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6680.585762] Lustre: Skipped 2 previous similar messages [ 6680.585964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6680.595338] Lustre: Skipped 6 previous similar messages [ 6690.272635] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6690.432967] Lustre: server umount lustre-OST0000 complete [ 6708.191844] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6708.379936] Lustre: server umount lustre-MDT0000 complete [ 6711.273582] LustreError: 172321:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6711.292302] LustreError: 172321:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 6712.011069] LustreError: 170340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868658 with bad export cookie 5069906498968304994 [ 6712.015643] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6712.019366] LustreError: 170340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6712.327704] Lustre: server umount lustre-MDT0001 complete [ 6720.605664] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 18:11:05 (1787868665) [ 6726.847973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6730.809299] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6738.505629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6744.305567] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6754.023739] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6760.354721] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6767.343533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6772.980126] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6779.046637] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6791.688301] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 18:12:16 (1787868736) [ 6797.284448] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6797.313135] LustreError: Skipped 1 previous similar message [ 6797.330911] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6797.355983] Lustre: Skipped 2 previous similar messages [ 6802.433321] Lustre: server umount lustre-OST0000 complete [ 6809.571885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6809.583396] Lustre: Skipped 13 previous similar messages [ 6811.701867] Lustre: server umount lustre-MDT0000 complete [ 6815.109808] LustreError: 174711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868761 with bad export cookie 5069906498968305876 [ 6815.112046] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6815.115844] LustreError: 174711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6815.446426] Lustre: server umount lustre-MDT0001 complete [ 6826.574893] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 6829.537748] Key type lgssc unregistered [ 6829.875445] LNet: 178461:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6829.885661] LNetError: 178461:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6829.922926] LNet: Removed LNI 192.168.203.109@tcp [ 6830.707288] Key type .llcrypt unregistered [ 6830.711481] Key type ._llcrypt unregistered [ 6848.742448] Key type ._llcrypt registered [ 6848.744849] Key type .llcrypt registered [ 6848.846284] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 6849.956036] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6849.981966] alg: No test for adler32 (adler32-zlib) [ 6851.038659] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 6851.309493] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 6852.983156] Key type lgssc registered [ 6854.330086] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6862.223960] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6862.245626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6877.792264] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6882.911368] LustreError: 179705:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.109@tcp: failed processing log, type 4: rc = -110 [ 6888.032496] LustreError: 179705:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6888.043733] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6888.054169] LustreError: MGC192.168.203.109@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6888.064679] LustreError: 179705:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6888.089701] LustreError: 179705:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 6888.101615] LustreError: 179705:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 6888.156725] Lustre: server umount lustre-OST0000 complete [ 6888.160479] LustreError: 179705:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6895.907455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6897.351755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6901.250877] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6907.877562] LustreError: 180204:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6907.904067] LustreError: 180204:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 6908.990143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6909.410808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6914.274734] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6922.484498] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6927.919711] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6934.721567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6934.947392] LustreError: 182019:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6934.954932] Lustre: 182019:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.109@tcp: invalid local config log lustre-OST0000: rc = -22 [ 6935.063461] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6942.064991] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6949.750018] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6953.444474] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6953.450614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6953.473502] Lustre: Skipped 2 previous similar messages [ 6953.500321] Lustre: Skipped 2 previous similar messages [ 6957.837530] Lustre: server umount lustre-MDT0000 complete [ 6958.563963] LustreError: 180205:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6958.585620] LustreError: 180205:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 6960.952850] LustreError: 180187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868907 with bad export cookie 325829640200152983 [ 6960.961085] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6960.969801] LustreError: 180187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6961.239711] Lustre: server umount lustre-MDT0001 complete [ 6974.903806] Lustre: server umount lustre-OST0000 complete [ 6983.397286] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 6992.634134] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 7026.730721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7026.936105] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7026.963116] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7027.084220] Lustre: lustre-MDT0000: new disk, initializing [ 7027.187858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7027.214320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7031.411075] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7041.315642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7041.443340] Lustre: 185546:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7041.494587] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7041.498392] Lustre: Skipped 1 previous similar message [ 7041.611273] Lustre: lustre-MDT0001: new disk, initializing [ 7041.799777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7041.845175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7041.866957] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7048.037626] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7059.885943] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7066.391097] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7073.601905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7073.877229] Lustre: lustre-OST0000: new disk, initializing [ 7073.881265] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7073.891777] Lustre: 187513:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7074.021465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7080.832616] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7082.570246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7082.577202] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7082.678844] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7090.242974] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7094.455140] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7094.628305] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7099.340269] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7099.562303] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7103.460180] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7103.461170] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7103.467726] LustreError: Skipped 1 previous similar message [ 7103.472656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7103.505702] Lustre: Skipped 1 previous similar message [ 7107.512525] Lustre: server umount lustre-OST0000 complete [ 7113.698317] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7113.702090] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7113.705416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7113.705423] Lustre: Skipped 1 previous similar message [ 7113.734209] Lustre: Skipped 1 previous similar message [ 7116.897322] Lustre: server umount lustre-MDT0000 complete [ 7118.816561] LustreError: 185552:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7118.837364] LustreError: 185552:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 7120.347079] LustreError: 185538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869067 with bad export cookie 325829640200153704 [ 7120.362438] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7120.367080] LustreError: 185538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7120.736331] Lustre: server umount lustre-MDT0001 complete [ 7128.077246] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 18:17:53 (1787869073) [ 7142.856993] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 7145.175604] Key type lgssc unregistered [ 7145.450194] LNet: 190459:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7145.456973] LNetError: 190459:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7145.476114] LNet: Removed LNI 192.168.203.109@tcp [ 7146.240875] Key type .llcrypt unregistered [ 7146.245835] Key type ._llcrypt unregistered [ 7165.006095] Key type ._llcrypt registered [ 7165.014361] Key type .llcrypt registered [ 7165.219695] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 7166.455429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7166.481371] alg: No test for adler32 (adler32-zlib) [ 7167.581879] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 7167.830790] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 7169.511279] Key type lgssc registered [ 7170.590833] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7177.272241] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7177.295433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7178.730367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7182.925927] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7188.963149] LustreError: 191682:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7188.979050] LustreError: 191682:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 7190.678101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7191.054398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7195.090513] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7201.977170] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7207.526239] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7214.886086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7215.093549] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7215.100532] LustreError: 193546:0:(llog.c:1696:llog_backup()) MGC192.168.203.109@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7215.105377] Lustre: 193546:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.109@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7215.175556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7222.205609] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7229.921546] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7234.025015] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7234.028273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7234.054938] Lustre: Skipped 2 previous similar messages [ 7234.066488] Lustre: Skipped 2 previous similar messages [ 7238.443377] Lustre: server umount lustre-MDT0000 complete [ 7239.141832] LustreError: 191678:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7239.179782] LustreError: 191678:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 7241.855128] LustreError: 191661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869188 with bad export cookie 7050271102730338703 [ 7241.856122] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7241.873121] LustreError: 191661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7242.263459] Lustre: server umount lustre-MDT0001 complete [ 7255.704229] Lustre: server umount lustre-OST0000 complete [ 7266.535452] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 7277.719695] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 7315.406601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7315.730857] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7315.790374] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7315.902923] Lustre: lustre-MDT0000: new disk, initializing [ 7316.034786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7316.073726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7321.210419] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7331.446424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7331.616687] Lustre: 197074:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7331.676477] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7331.679717] Lustre: Skipped 1 previous similar message [ 7331.682583] LustreError: 197855:0:(llog.c:1696:llog_backup()) MGC192.168.203.109@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7331.688721] LustreError: 197855:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7331.693088] Lustre: 197855:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.109@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7331.713858] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7331.719817] Lustre: Skipped 1 previous similar message [ 7331.911615] Lustre: lustre-MDT0001: new disk, initializing [ 7332.044817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7332.100911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7332.112787] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7336.981342] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7347.883157] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7354.268831] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7361.572158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7361.933924] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7361.939046] Lustre: Skipped 1 previous similar message [ 7361.942604] LustreError: 199016:0:(llog.c:1696:llog_backup()) MGC192.168.203.109@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7361.951450] LustreError: 199016:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7361.957684] Lustre: 199016:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.109@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7361.976452] Lustre: 199016:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7362.015904] Lustre: lustre-OST0000: new disk, initializing [ 7362.021819] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7362.034950] Lustre: 199040:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7362.166263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7368.259642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7368.271888] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7368.393329] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7369.034570] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7377.512596] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7382.688410] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7382.942879] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7387.784224] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7388.046447] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7394.281288] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7394.287171] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7394.300079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7394.317342] Lustre: Skipped 1 previous similar message [ 7394.331884] Lustre: Skipped 1 previous similar message [ 7396.111642] Lustre: server umount lustre-OST0000 complete [ 7404.000298] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7404.008280] LustreError: Skipped 1 previous similar message [ 7404.024136] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7404.048132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7409.638477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7409.649244] Lustre: Skipped 3 previous similar messages [ 7414.239579] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7414.388744] Lustre: server umount lustre-MDT0000 complete [ 7414.753131] LustreError: 197085:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7414.771269] LustreError: 197085:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 7417.401284] LustreError: 199039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869364 with bad export cookie 7050271102730339424 [ 7417.402582] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7417.415479] LustreError: 199039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7417.711559] Lustre: server umount lustre-MDT0001 complete [ 7425.278792] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 18:22:50 (1787869370) [ 7440.427394] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 7449.530208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7449.871428] LustreError: 202168:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7449.906899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7454.653902] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7455.205698] LustreError: 202169:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7463.059734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7468.329372] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7471.605367] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7479.007791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7485.601616] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7494.142247] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7494.356426] Lustre: lustre-OST0001: new disk, initializing [ 7494.368798] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7494.377404] Lustre: 204478:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7494.473246] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7494.486051] Lustre: Skipped 2 previous similar messages [ 7498.872118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7498.891658] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7498.995963] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7500.546873] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7511.523608] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7522.381958] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7524.040605] Lustre: DEBUG MARKER: ADD OST3 [ 7528.349139] loop: module loaded [ 7528.419539] loop0: detected capacity change from 0 to 204800000 [ 7528.596224] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7528.606502] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7528.622463] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7534.678753] loop0: detected capacity change from 0 to 204800000 [ 7535.483186] Lustre: DEBUG MARKER: STOP OST3 [ 7540.361574] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7540.504290] Lustre: lustre-OST0003: new disk, initializing [ 7540.509096] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7540.515266] Lustre: 206470:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7542.368266] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7542.492470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7542.495818] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7542.536526] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7544.415321] LustreError: 206600:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7545.226611] Lustre: Failing over lustre-OST0003 [ 7545.339866] Lustre: server umount lustre-OST0003 complete [ 7547.172883] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7547.539918] LustreError: 203656:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7547.553787] LustreError: 203656:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 7547.882784] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7547.884040] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7547.892993] Lustre: Skipped 1 previous similar message [ 7555.351323] Lustre: DEBUG MARKER: START OST3 again [ 7561.751249] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7562.004492] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7562.014306] Lustre: Skipped 1 previous similar message [ 7562.024177] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7562.870292] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7563.225713] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7563.229952] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7567.535371] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7587.252597] Lustre: Failing over lustre-OST0003 [ 7587.370474] Lustre: server umount lustre-OST0003 complete [ 7587.826162] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7587.841594] Lustre: Skipped 1 previous similar message [ 7587.848850] LustreError: 204747:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7587.873617] LustreError: 204747:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 9 previous similar messages [ 7606.762914] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7606.780739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7606.804542] Lustre: Skipped 1 previous similar message [ 7606.814730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7611.640388] Lustre: server umount lustre-MDT0000 complete [ 7615.187741] LustreError: 202148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869561 with bad export cookie 7050271102730340306 [ 7615.206811] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7615.209404] LustreError: 202148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7618.532221] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7618.542277] Lustre: Skipped 5 previous similar messages [ 7620.577309] LustreError: 203366:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7620.608409] LustreError: 203366:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 15 previous similar messages [ 7621.635252] Lustre: server umount lustre-MDT0001 complete [ 7625.293106] Lustre: server umount lustre-OST0000 complete [ 7629.663937] Lustre: server umount lustre-OST0001 complete [ 7636.453468] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 7645.391656] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 7679.782193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7679.980220] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7680.032433] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7680.115642] Lustre: lustre-MDT0000: new disk, initializing [ 7680.245823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7684.281375] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7693.975392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7694.116324] Lustre: 211936:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7694.389577] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7694.402600] Lustre: Skipped 1 previous similar message [ 7694.447115] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7698.981406] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7707.985962] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7713.219377] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7719.972829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7720.360212] Lustre: lustre-OST0000: new disk, initializing [ 7720.363438] Lustre: Skipped 1 previous similar message [ 7720.368929] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7720.375536] Lustre: Skipped 2 previous similar messages [ 7720.380960] Lustre: 213903:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7722.176643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7722.191482] Lustre: Skipped 1 previous similar message [ 7722.200477] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7722.348299] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7727.186840] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7736.690553] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7741.205849] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7741.446993] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7746.534449] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7746.778594] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7753.190375] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7753.194471] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7753.201028] LustreError: Skipped 1 previous similar message [ 7753.203385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7753.215697] Lustre: Skipped 6 previous similar messages [ 7753.224181] Lustre: Skipped 2 previous similar messages [ 7754.880444] Lustre: server umount lustre-OST0000 complete [ 7764.456245] Lustre: server umount lustre-MDT0000 complete [ 7768.101131] LustreError: 211927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869714 with bad export cookie 7050271102730342602 [ 7768.117985] LustreError: 211927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7768.124479] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7768.485756] Lustre: server umount lustre-MDT0001 complete [ 7772.171934] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 18:28:36 (1787869716) [ 7788.139487] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 7798.633644] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 7833.785259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7834.003515] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7834.033458] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7834.122897] Lustre: lustre-MDT0000: new disk, initializing [ 7834.193347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7838.271324] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7848.947324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7849.097657] Lustre: 218516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7849.409639] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7854.226029] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7863.085116] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7868.000354] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7874.460259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7874.689634] Lustre: 220485:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7876.787375] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7881.267880] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7890.776969] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7895.385247] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7895.595810] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7900.023544] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7900.199680] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7902.178900] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7902.190894] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7902.205968] Lustre: Skipped 2 previous similar messages [ 7902.218377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7902.227309] Lustre: Skipped 4 previous similar messages [ 7908.300358] Lustre: server umount lustre-OST0000 complete [ 7917.802426] Lustre: server umount lustre-MDT0000 complete [ 7918.069323] LustreError: 218523:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7918.093444] LustreError: 218523:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 7922.798718] LustreError: 218506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869869 with bad export cookie 7050271102730343484 [ 7922.800595] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7922.821027] LustreError: 218506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7923.980021] Lustre: server umount lustre-MDT0001 complete [ 7937.103546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7944.102918] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7952.741194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7953.174200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7953.182800] Lustre: Skipped 5 previous similar messages [ 7957.054686] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7964.977132] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7970.841786] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7976.931448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7983.005161] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7992.906779] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8004.065049] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8004.065196] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8004.083918] LustreError: Skipped 2 previous similar messages [ 8004.089286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8004.089294] Lustre: Skipped 5 previous similar messages [ 8004.149755] Lustre: Skipped 4 previous similar messages [ 8009.881455] Lustre: server umount lustre-OST0000 complete [ 8019.783543] Lustre: server umount lustre-MDT0000 complete [ 8023.574606] LustreError: 224544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869970 with bad export cookie 7050271102730344366 [ 8023.586665] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8024.051871] Lustre: server umount lustre-MDT0001 complete [ 8133.283686] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 8136.262332] Key type lgssc unregistered [ 8136.594376] LNet: 226368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8136.628555] LNetError: 226368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8136.665640] LNet: Removed LNI 192.168.203.109@tcp [ 8137.669224] Key type .llcrypt unregistered [ 8137.672832] Key type ._llcrypt unregistered [ 8166.177105] Key type ._llcrypt registered [ 8166.182249] Key type .llcrypt registered [ 8166.335925] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 8183.614479] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 8184.739590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8184.808802] alg: No test for adler32 (adler32-zlib) [ 8186.093525] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 8186.475271] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 8188.240200] Key type lgssc registered [ 8189.839243] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8225.172031] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8225.213415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8226.599800] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8226.643097] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8226.792472] Lustre: lustre-MDT0000: new disk, initializing [ 8226.888611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8226.911320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8231.577834] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8243.128806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8243.247412] Lustre: 229723:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8243.322414] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8243.326091] Lustre: Skipped 1 previous similar message [ 8243.394104] Lustre: lustre-MDT0001: new disk, initializing [ 8243.533435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8243.564171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8243.573622] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8248.533860] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8258.228252] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8263.600968] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8270.072909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8270.518567] Lustre: lustre-OST0000: new disk, initializing [ 8270.532489] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8270.566214] Lustre: 231691:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8270.741299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8276.062125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8276.077027] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8276.236064] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8277.130404] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8286.350893] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8291.979205] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8292.209511] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8296.894801] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8297.130638] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8302.052786] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8302.057878] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8302.089124] Lustre: Skipped 1 previous similar message [ 8302.091411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8307.180288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8307.188981] Lustre: Skipped 2 previous similar messages [ 8312.306180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8312.316816] Lustre: Skipped 1 previous similar message [ 8313.311433] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8313.524776] Lustre: server umount lustre-OST0000 complete [ 8317.409929] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8317.416969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8317.424738] Lustre: Skipped 1 previous similar message [ 8317.445910] Lustre: Skipped 1 previous similar message [ 8322.536178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8322.546362] Lustre: Skipped 1 previous similar message [ 8323.782349] Lustre: server umount lustre-MDT0000 complete [ 8327.683279] LustreError: 229715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870274 with bad export cookie 7162756804129847383 [ 8327.689527] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8327.705853] LustreError: 229715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8328.162968] Lustre: server umount lustre-MDT0001 complete [ 8348.207321] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 8359.517877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8359.982788] LustreError: 234624:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8359.998634] LustreError: 234624:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 8360.043956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8364.618977] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8365.538921] LustreError: 234625:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8365.558169] LustreError: 234625:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 8369.640746] LustreError: 234624:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8372.977867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8373.222251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8377.258639] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8380.446194] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8388.834582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8389.623849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8397.863851] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8406.463516] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8406.742134] Lustre: lustre-OST0001: new disk, initializing [ 8406.748535] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8406.767554] Lustre: 236933:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8406.891767] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8412.211559] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8414.794928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8414.821850] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8414.962943] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8423.326807] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8430.191743] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8444.580678] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 18:39:49 (1787870389) [ 8450.016407] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8450.033388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8450.055499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8453.020098] Lustre: server umount lustre-MDT0000 complete [ 8456.166169] LustreError: 234620:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8456.190418] LustreError: 234620:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 8456.887931] LustreError: 234605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870403 with bad export cookie 7162756804129848265 [ 8456.899952] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8456.903856] LustreError: 234605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8457.251847] Lustre: server umount lustre-MDT0001 complete [ 8470.643794] Lustre: server umount lustre-OST0000 complete [ 8485.203598] Lustre: server umount lustre-OST0001 complete [ 8492.820297] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 8504.689034] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 8547.021445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8547.281773] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8547.310296] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8547.423777] Lustre: lustre-MDT0000: new disk, initializing [ 8547.554915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8547.582843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8552.420519] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8562.602716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8562.736278] Lustre: 240882:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8562.799379] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8562.805372] Lustre: Skipped 1 previous similar message [ 8562.958779] Lustre: lustre-MDT0001: new disk, initializing [ 8563.114744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8563.127241] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8567.025108] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8579.210954] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8586.719234] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8593.747408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8594.112849] Lustre: lustre-OST0000: new disk, initializing [ 8594.116995] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8594.147718] Lustre: 242853:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8595.459934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8595.466248] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8595.521660] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8600.468318] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8610.478457] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8615.491568] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8615.731521] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8620.915807] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8621.149984] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8626.144659] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8626.149291] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8626.156416] LustreError: Skipped 1 previous similar message [ 8626.157991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8626.175944] Lustre: Skipped 4 previous similar messages [ 8626.200044] Lustre: Skipped 5 previous similar messages [ 8629.376684] Lustre: server umount lustre-OST0000 complete [ 8634.856099] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8634.862860] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8647.135333] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8647.359822] Lustre: server umount lustre-MDT0000 complete [ 8650.728880] LustreError: 240889:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8650.783450] LustreError: 240889:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 8651.637923] LustreError: 240874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870598 with bad export cookie 7162756804129850400 [ 8651.642625] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8651.654816] LustreError: 240874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8652.004703] Lustre: server umount lustre-MDT0001 complete [ 8661.105647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8661.649458] LustreError: 245069:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8661.714100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8661.720852] Lustre: Skipped 2 previous similar messages [ 8666.648981] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8676.739928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8681.943791] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8690.173399] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8695.580646] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8701.347611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8708.034767] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8716.130301] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8863.722653] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8863.724774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8863.745359] Lustre: Skipped 2 previous similar messages [ 8863.760739] Lustre: Skipped 8 previous similar messages [ 8867.904110] Lustre: server umount lustre-OST0000 complete [ 8871.906674] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8886.239558] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8886.407374] Lustre: server umount lustre-MDT0000 complete [ 8889.318854] LustreError: 245065:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8889.343536] LustreError: 245065:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 8890.113573] LustreError: 245050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870836 with bad export cookie 7162756804129851282 [ 8890.115261] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8890.126322] LustreError: 245050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8890.581747] Lustre: server umount lustre-MDT0001 complete [ 8986.751814] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 8989.864818] Key type lgssc unregistered [ 8990.261545] LNet: 248683:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8990.274477] LNetError: 248683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8990.297354] LNet: Removed LNI 192.168.203.109@tcp [ 8991.248166] Key type .llcrypt unregistered [ 8991.251330] Key type ._llcrypt unregistered [ 9017.055123] Key type ._llcrypt registered [ 9017.058467] Key type .llcrypt registered [ 9017.176990] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 9033.703887] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 9035.026482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9035.131978] alg: No test for adler32 (adler32-zlib) [ 9036.384607] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 9036.651049] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 9038.351935] Key type lgssc registered [ 9039.543652] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9084.865460] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 9095.512602] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9095.547147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9096.895248] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9096.921183] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9097.026759] Lustre: lustre-MDT0000: new disk, initializing [ 9097.158473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9097.195210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9101.889396] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9112.367091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9112.453757] Lustre: 252756:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9112.505916] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9112.510880] Lustre: Skipped 1 previous similar message [ 9112.593206] Lustre: lustre-MDT0001: new disk, initializing [ 9112.722774] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9112.770927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9112.791479] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9117.858184] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9122.673338] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9129.352385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9129.678375] Lustre: lustre-OST0000: new disk, initializing [ 9129.683414] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9129.694789] Lustre: 254386:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9129.783257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9130.063654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9130.085470] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9130.298410] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9136.425872] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9146.159798] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9146.314859] Lustre: lustre-OST0001: new disk, initializing [ 9146.318322] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9146.323972] Lustre: 255256:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9146.427191] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9152.144418] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9152.607982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9152.626598] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9152.766630] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9162.811517] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9170.446271] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9183.242816] Lustre: server umount lustre-MDT0000 complete [ 9183.729446] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9183.752174] Lustre: Skipped 1 previous similar message [ 9186.615266] LustreError: 255255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871133 with bad export cookie 9368131411078853457 [ 9186.622361] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9186.631127] LustreError: 255255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9186.983381] Lustre: server umount lustre-MDT0001 complete [ 9200.702323] Lustre: server umount lustre-OST0000 complete [ 9214.854505] Lustre: server umount lustre-OST0001 complete [ 9227.468392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9235.485831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9244.421893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9253.473556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9272.618601] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 9284.569119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9284.591602] Lustre: lustre-MDT0000: reset Object Index mappings [ 9285.070157] LustreError: 260268:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9285.086930] LustreError: 260268:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 5 previous similar messages [ 9285.168386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9289.695036] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9290.209897] LustreError: 260269:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9294.307406] LustreError: 260268:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9298.715030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9298.740626] Lustre: lustre-MDT0001: reset Object Index mappings [ 9299.439247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9304.379195] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9307.555364] Lustre: 261411:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9313.847719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9319.827885] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9324.470217] LustreError: 261765:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9327.583931] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9327.855750] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9327.858808] Lustre: Skipped 1 previous similar message [ 9329.914396] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9329.940242] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9333.385580] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9340.500749] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9343.621229] Lustre: 263283:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9350.630836] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9350.640663] LustreError: 260265:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9350.648845] Lustre: Skipped 4 previous similar messages [ 9350.669273] LustreError: 260265:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 9350.679327] Lustre: server umount lustre-MDT0000 complete [ 9354.784777] LustreError: 261012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871301 with bad export cookie 9368131411078860247 [ 9354.790630] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9354.797922] LustreError: 261012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9355.182949] Lustre: server umount lustre-MDT0001 complete [ 9370.230914] Lustre: server umount lustre-OST0000 complete [ 9372.127141] Lustre: 250262:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787871302/real 1787871302] req@ffff92f146ed2940 x1874718603993600/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787871318 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9372.180057] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9372.203753] Lustre: Skipped 1 previous similar message [ 9373.874842] Lustre: server umount lustre-OST0001 complete [ 9391.327761] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 18:55:36 (1787871336) [ 9406.385936] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 9415.650979] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 9452.273433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9452.482123] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9452.513558] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9452.609660] Lustre: lustre-MDT0000: new disk, initializing [ 9452.714118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9452.728239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9457.363071] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9466.597218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9466.709336] Lustre: 267532:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9466.756923] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9466.760474] Lustre: Skipped 1 previous similar message [ 9466.821567] Lustre: lustre-MDT0001: new disk, initializing [ 9466.954679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9466.978491] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9471.742590] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9480.419806] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9485.996168] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9492.811225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9493.148662] Lustre: lustre-OST0000: new disk, initializing [ 9493.153219] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9493.160906] Lustre: 269503:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9494.469511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9494.482168] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9494.669925] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9500.188825] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9509.665410] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9514.620842] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9514.837654] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9518.676891] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9518.882868] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9525.217056] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9525.220815] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9525.230120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9525.249920] Lustre: Skipped 1 previous similar message [ 9525.263410] Lustre: Skipped 1 previous similar message [ 9527.330818] Lustre: server umount lustre-OST0000 complete [ 9533.410301] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9533.424168] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9533.434194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9535.460967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9535.479067] Lustre: Skipped 1 previous similar message [ 9537.539185] Lustre: server umount lustre-MDT0000 complete [ 9540.576580] LustreError: 267545:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9540.584862] LustreError: 267545:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 9541.151061] LustreError: 269502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871487 with bad export cookie 9368131411078862340 [ 9541.161624] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9541.166800] LustreError: 269502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9541.743198] Lustre: server umount lustre-MDT0001 complete [ 9559.660911] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 9568.454737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9568.711289] LustreError: 272437:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9568.745200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9568.749210] Lustre: Skipped 2 previous similar messages [ 9572.178725] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9580.133069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9585.059893] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9587.999896] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9594.117170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9600.675549] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9609.503692] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9609.698477] Lustre: lustre-OST0001: new disk, initializing [ 9609.700934] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9609.713611] Lustre: 274743:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9611.146320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9611.162204] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9611.325111] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9617.876968] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9628.533571] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9631.603874] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9636.461651] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9636.667856] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9645.029095] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9645.034175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9645.043752] Lustre: Skipped 3 previous similar messages [ 9645.057471] Lustre: Skipped 3 previous similar messages [ 9649.766214] Lustre: server umount lustre-MDT0000 complete [ 9650.144914] LustreError: 273165:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9650.170547] LustreError: 273165:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 5 previous similar messages [ 9653.533653] LustreError: 272417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871600 with bad export cookie 9368131411078863222 [ 9653.543902] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9653.549136] LustreError: 272417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9653.886857] Lustre: server umount lustre-MDT0001 complete [ 9669.238591] Lustre: server umount lustre-OST0000 complete [ 9670.367216] Lustre: 250262:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787871601/real 1787871601] req@ffff92f14972f840 x1874718604104960/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787871617 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9670.394347] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9670.404468] Lustre: Skipped 1 previous similar message [ 9673.218694] Lustre: server umount lustre-OST0001 complete [ 9688.641914] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 9699.553093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9700.066989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9700.073519] Lustre: Skipped 3 previous similar messages [ 9704.438595] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9712.384374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9718.554341] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9721.329671] Lustre: 278952:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9727.680559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9734.367635] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9738.289841] LustreError: 279306:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9738.329700] LustreError: 279306:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 9742.960776] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9744.354067] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9744.386337] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9749.259604] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9756.404678] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9759.621377] Lustre: 280830:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9764.323636] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9764.395645] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9772.491175] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 19:01:57 (1787871717) [ 9789.921822] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9789.929985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9789.951319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9793.347519] Lustre: server umount lustre-MDT0000 complete [ 9798.253658] LustreError: 277780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871744 with bad export cookie 9368131411078870019 [ 9798.255997] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9798.272197] LustreError: 277780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9800.677363] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9800.683280] Lustre: Skipped 5 previous similar messages [ 9804.697804] Lustre: server umount lustre-MDT0001 complete [ 9822.178325] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9822.278483] Lustre: server umount lustre-OST0000 complete [ 9840.095638] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9840.574405] Lustre: server umount lustre-OST0001 complete [ 9848.290389] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 9858.900735] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 9903.228571] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 9913.269807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9913.488773] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9913.541224] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9913.653311] Lustre: lustre-MDT0000: new disk, initializing [ 9913.788609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9918.768549] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9930.603848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9930.791217] Lustre: 285072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9931.156566] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9936.015851] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9947.013725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9947.327922] Lustre: 286707:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9949.070964] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9953.510235] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9964.577804] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9964.729984] Lustre: 287579:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9964.828940] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9964.838386] Lustre: Skipped 6 previous similar messages [ 9965.932649] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9965.952131] Lustre: Skipped 1 previous similar message [ 9966.130319] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9971.555482] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9981.908039] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9985.494073] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9985.496822] Lustre: Skipped 1 previous similar message [10000.431268] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 19:05:45 (1787871945) [10002.912054] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10002.915355] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10002.931489] Lustre: Skipped 5 previous similar messages [10002.936634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10002.957902] Lustre: Skipped 2 previous similar messages [10004.823456] Lustre: server umount lustre-MDT0000 complete [10007.017967] LustreError: 285086:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10007.049199] LustreError: 285086:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 11 previous similar messages [10008.487368] LustreError: 285067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871955 with bad export cookie 9368131411078877439 [10008.491452] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10008.502309] LustreError: 285067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10008.851596] Lustre: server umount lustre-MDT0001 complete [10023.188201] Lustre: server umount lustre-OST0000 complete [10037.224904] Lustre: server umount lustre-OST0001 complete [10044.834394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10050.113985] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10059.426915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10065.023506] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10073.213316] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10079.260596] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10086.340467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10093.566986] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10101.918551] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10123.111527] loop1: detected capacity change from 0 to 204800000 [10123.316698] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10123.341593] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10123.373105] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10129.270991] loop1: detected capacity change from 0 to 204800000 [10135.312072] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10135.419574] LustreError: 289813:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs32655: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10135.427990] LustreError: 289813:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs32655 [10135.433374] LustreError: 289813:0:(mgs_handler.c:517:mgs_target_reg()) fs32655-MDT0001: Error checking target index: rc = -13 [10135.441504] LustreError: fs32655-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10135.449616] LustreError: 293546:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10135.457960] LustreError: 293546:0:(tgt_mount.c:2129:server_put_super()) no obd fs32655-MDT0001 [10135.465426] LustreError: 293546:0:(tgt_mount.c:116:server_deregister_mount()) fs32655-MDT0001 not registered [10135.518327] Lustre: server umount fs32655-MDT0001 complete [10135.521323] LustreError: 293546:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10143.093767] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10143.168696] Lustre: Setting parameter fs32655-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs32655-MDT0001 [10143.205350] Lustre: srv-fs32655-MDT0001: No data found on store. Initialize space. [10143.208024] Lustre: Skipped 4 previous similar messages [10143.261261] Lustre: fs32655-MDT0001: new disk, initializing [10143.264226] Lustre: Skipped 3 previous similar messages [10143.367397] Lustre: srv-fs32655-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10143.375536] Lustre: 294012:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs32655-MDT0001-mdtlov: can't create new llog: rc = -115 [10143.386810] LustreError: 294012:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs32655-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10148.391663] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10156.377666] Lustre: server umount fs32655-MDT0001 complete [10170.721604] loop1: detected capacity change from 0 to 204800000 [10170.870212] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10176.932786] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10177.505893] Lustre: 295583:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs32655-MDT0001-mdtlov: can't create new llog: rc = -115 [10177.513619] LustreError: 295583:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs32655-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10182.473250] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10186.209142] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10187.774475] Lustre: server umount fs32655-MDT0001 complete [10200.230757] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.9@tcp (stopping) [10201.582496] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10201.599559] Lustre: Skipped 5 previous similar messages [10204.899712] Lustre: server umount lustre-MDT0000 complete [10214.368962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10214.499689] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10220.012128] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10220.078288] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10228.273882] loop1: detected capacity change from 0 to 204800000 [10234.774274] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10235.348143] Lustre: 298032:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs32655-MDT0001-mdtlov: can't create new llog: rc = -115 [10235.358581] LustreError: 298032:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs32655-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10240.097354] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10245.262977] Lustre: server umount fs32655-MDT0001 complete [10266.096148] LustreError: 291657:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10266.132889] LustreError: 291657:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 108 previous similar messages [10271.204569] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10271.213540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10271.225560] Lustre: Skipped 4 previous similar messages [10275.394833] Lustre: server umount lustre-OST0000 complete [10285.925640] Lustre: server umount lustre-MDT0000 complete [10290.127613] LustreError: 289806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872236 with bad export cookie 9368131411078880120 [10290.144697] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10290.530750] Lustre: server umount lustre-MDT0001 complete [10303.962194] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [10307.606397] Key type lgssc unregistered [10307.985610] LNet: 300379:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10307.990810] LNetError: 300379:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10308.041572] LNet: Removed LNI 192.168.203.109@tcp [10309.178164] Key type .llcrypt unregistered [10309.190903] Key type ._llcrypt unregistered [10329.451684] Key type ._llcrypt registered [10329.453865] Key type .llcrypt registered [10329.541078] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [10331.293911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10331.519274] alg: No test for adler32 (adler32-zlib) [10332.905742] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [10333.299040] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [10335.119759] Key type lgssc registered [10336.719158] Lustre: Echo OBD driver; http://www.lustre.org/ [10346.522786] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10346.536890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10348.074238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10352.904504] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10358.253938] LustreError: 301655:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10358.289953] LustreError: 301655:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [10362.416631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10362.986710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10368.074151] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10376.358950] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10382.390487] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10389.973624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10390.482434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10392.548434] LustreError: 303484:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10397.279905] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10397.676216] LustreError: 303485:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10402.794778] LustreError: 303485:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10405.421932] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10407.910810] LustreError: 303486:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10418.156204] LustreError: 303609:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10418.182582] LustreError: 303609:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 6 previous similar messages [10424.605878] loop1: detected capacity change from 0 to 204800000 [10424.777629] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10424.809117] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10424.819783] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10430.053168] loop1: detected capacity change from 0 to 204800000 [10434.164566] LustreError: 303609:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10434.197214] LustreError: 303609:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 9 previous similar messages [10435.217501] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10435.364313] Lustre: fs2077-OST0000: new disk, initializing [10435.367854] Lustre: srv-fs2077-OST0000: No data found on store. Initialize space. [10435.374662] Lustre: 305390:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10435.438404] Lustre: fs2077-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10441.883749] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10450.560767] Lustre: server umount fs2077-OST0000 complete [10462.943430] loop1: detected capacity change from 0 to 204800000 [10468.758368] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10469.027302] Lustre: fs2077-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10469.348434] LustreError: 303486:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10469.380402] LustreError: 303486:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 19 previous similar messages [10474.486165] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10474.996951] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10483.717835] loop2: detected capacity change from 0 to 204800000 [10483.913541] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10483.932823] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10483.962056] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10489.319148] loop2: detected capacity change from 0 to 204800000 [10489.854803] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10495.197293] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10495.307389] LustreError: fs2077-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10495.314700] LustreError: fs2077-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10495.337247] LustreError: 308432:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10495.342827] LustreError: 308432:0:(tgt_mount.c:2129:server_put_super()) no obd fs2077-OST0001 [10495.352174] LustreError: 308432:0:(tgt_mount.c:116:server_deregister_mount()) fs2077-OST0001 not registered [10495.401381] Lustre: server umount fs2077-OST0001 complete [10495.406464] LustreError: 308432:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10505.388981] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10505.487329] Lustre: fs2077-OST0001: new disk, initializing [10505.490791] Lustre: srv-fs2077-OST0001: No data found on store. Initialize space. [10505.496367] Lustre: 308984:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10505.579095] Lustre: fs2077-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10511.808894] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10520.943201] Lustre: server umount fs2077-OST0000 complete [10530.613631] Lustre: server umount fs2077-OST0001 complete [10535.922128] LustreError: 303609:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10535.945417] LustreError: 303609:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 38 previous similar messages [10543.035272] loop1: detected capacity change from 0 to 204800000 [10548.618087] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10549.074445] Lustre: fs2077-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10555.561958] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10562.510601] loop2: detected capacity change from 0 to 204800000 [10567.911541] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10575.222172] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10585.317701] Lustre: server umount fs2077-OST0000 complete [10595.343622] Lustre: server umount fs2077-OST0001 complete [10612.704334] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10612.705352] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10612.711290] LustreError: Skipped 1 previous similar message [10612.719644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10612.735425] Lustre: Skipped 1 previous similar message [10614.388877] Lustre: server umount lustre-OST0000 complete [10618.848215] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10618.871015] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10618.895682] Lustre: Skipped 1 previous similar message [10618.910166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10622.946967] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10622.950626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10622.999477] Lustre: Skipped 1 previous similar message [10624.760902] Lustre: server umount lustre-MDT0000 complete [10628.974789] LustreError: 301635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872575 with bad export cookie 12842177472827251396 [10628.999030] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10629.016253] LustreError: 301635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10629.676167] Lustre: server umount lustre-MDT0001 complete [10641.879947] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [10644.534099] Key type lgssc unregistered [10644.782812] LNet: 314911:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10644.789822] LNetError: 314911:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10645.867161] LNet: Removed LNI 192.168.203.109@tcp [10646.733243] Key type .llcrypt unregistered [10646.737797] Key type ._llcrypt unregistered [10681.929921] Key type ._llcrypt registered [10681.932090] Key type .llcrypt registered [10682.092085] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [10699.676071] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [10700.651603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10700.679391] alg: No test for adler32 (adler32-zlib) [10701.853420] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [10702.171024] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [10703.871217] Key type lgssc registered [10705.108616] Lustre: Echo OBD driver; http://www.lustre.org/ [10742.313742] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10742.352160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10743.593284] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10743.617852] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10743.684869] Lustre: lustre-MDT0000: new disk, initializing [10743.794297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10743.813187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10748.560670] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10758.560282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10758.792813] Lustre: 318770:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10758.879842] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10758.882839] Lustre: Skipped 1 previous similar message [10758.950139] Lustre: lustre-MDT0001: new disk, initializing [10759.037249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10759.087211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10759.098325] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10763.939323] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10773.276684] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10777.681930] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10785.555993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10785.979833] Lustre: lustre-OST0000: new disk, initializing [10785.990873] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10785.999914] Lustre: 320739:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10786.148647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10793.105752] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10793.575697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10793.611768] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10793.745799] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10802.505711] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10804.707235] Lustre: Mounted lustre-client - version 2.17.57_87_ge30c52f [10805.689643] Lustre: Unmounted lustre-client [10814.435564] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10814.443902] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10814.456855] LustreError: Skipped 1 previous similar message [10814.462264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10814.469366] Lustre: Skipped 1 previous similar message [10814.499688] Lustre: Skipped 1 previous similar message [10820.090732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10820.103916] Lustre: Skipped 1 previous similar message [10820.437525] Lustre: server umount lustre-OST0000 complete [10825.184687] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10825.193591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10825.203342] Lustre: Skipped 1 previous similar message [10830.306715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10830.313933] Lustre: Skipped 2 previous similar messages [10831.035373] Lustre: server umount lustre-MDT0000 complete [10834.944252] LustreError: 318761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872781 with bad export cookie 8927187634624245716 [10834.947780] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10834.962272] LustreError: 318761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10835.359871] Lustre: server umount lustre-MDT0001 complete [10846.573774] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [10849.426936] Key type lgssc unregistered [10859.871567] LNet: 1 local NIs in recovery (showing 1): 192.168.203.109@tcp [10859.936341] LNet: 322903:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10859.947627] LNetError: 322903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10933.735316] LNet: Removed LNI 192.168.203.109@tcp [10934.707432] Key type .llcrypt unregistered [10934.710743] Key type ._llcrypt unregistered [10955.067396] Key type ._llcrypt registered [10955.076853] Key type .llcrypt registered [10955.181976] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [10971.459648] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [10972.709556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10972.750437] alg: No test for adler32 (adler32-zlib) [10973.831190] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [10974.074570] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [10975.727168] Key type lgssc registered [10976.906162] Lustre: Echo OBD driver; http://www.lustre.org/ [11004.179853] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 19:22:29 (1787872949) [11010.144160] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11010.227536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11011.474869] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11011.519698] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11011.603325] Lustre: lustre-MDT0000: new disk, initializing [11011.684057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11011.699921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11015.701608] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11025.484540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11025.567736] Lustre: 326125:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11025.609839] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11025.613508] Lustre: Skipped 1 previous similar message [11025.665845] Lustre: lustre-MDT0001: new disk, initializing [11025.739517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11025.778405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11025.792099] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11030.179716] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11039.047331] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11044.381446] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11050.154559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11050.442725] Lustre: lustre-OST0000: new disk, initializing [11050.453089] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11050.462038] Lustre: 328093:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11050.572289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11056.233323] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11060.273027] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11060.292641] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11060.382321] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11064.710852] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11081.185483] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11081.188500] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11081.195433] LustreError: Skipped 1 previous similar message [11081.197304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11081.219089] Lustre: Skipped 1 previous similar message [11081.242707] Lustre: Skipped 1 previous similar message [11084.069820] Lustre: server umount lustre-OST0000 complete [11087.337222] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11087.345978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11087.376810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11091.426336] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11091.429100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11091.458909] Lustre: Skipped 1 previous similar message [11093.767601] Lustre: server umount lustre-MDT0000 complete [11097.980376] LustreError: 326115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873044 with bad export cookie 2898430723981045341 [11097.993808] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11097.998477] LustreError: 326115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11098.456434] Lustre: server umount lustre-MDT0001 complete [11112.596360] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [11116.934411] Key type lgssc unregistered [11117.268851] LNet: 330004:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11117.275328] LNetError: 330004:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11117.295813] LNet: Removed LNI 192.168.203.109@tcp [11118.318623] Key type .llcrypt unregistered [11118.327632] Key type ._llcrypt unregistered [11140.182859] Key type ._llcrypt registered [11140.184465] Key type .llcrypt registered [11140.358655] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [11141.433970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11141.554435] alg: No test for adler32 (adler32-zlib) [11142.814261] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [11143.141790] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [11144.825640] Key type lgssc registered [11145.935486] Lustre: Echo OBD driver; http://www.lustre.org/ [11153.515908] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11153.565780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11154.931088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11159.158609] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11165.154313] LustreError: 331281:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11165.172936] LustreError: 331281:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [11166.589038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11166.997856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11170.871874] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11177.158411] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11182.194370] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11187.729905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11188.129306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11194.446388] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11201.325064] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11210.323767] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11211.776406] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11218.402957] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11218.416929] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11218.432832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11219.938964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11219.942490] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11219.947688] Lustre: Skipped 1 previous similar message [11221.348976] Lustre: server umount lustre-MDT0000 complete [11223.676610] LustreError: 331275:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11224.588518] LustreError: 331260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873171 with bad export cookie 6277993999119509195 [11224.604093] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11225.056791] LustreError: 331277:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11225.060194] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11225.086492] Lustre: Skipped 1 previous similar message [11225.097242] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11225.108787] Lustre: Skipped 2 previous similar messages [11228.782783] LustreError: 331276:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11228.835826] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.9@tcp (stopping) [11230.966102] Lustre: server umount lustre-MDT0001 complete [11234.883284] Lustre: server umount lustre-OST0000 complete [11244.421871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11249.840401] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11252.532345] Lustre: 335460:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.109@tcp [11255.019394] Lustre: server umount MGS complete [11262.626830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11263.189333] LustreError: 336032:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11263.200014] LustreError: 336032:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [11263.258388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11268.230052] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11272.673939] LustreError: 336033:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11272.686467] LustreError: 336033:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [11276.694619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11277.167186] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11281.641089] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11290.534361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11290.987584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11297.403754] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11304.971212] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11306.645138] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11312.205515] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11313.931994] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11321.238567] Lustre: server umount lustre-OST0000 complete [11327.971301] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11327.984384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11327.990965] Lustre: Skipped 1 previous similar message [11338.209889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11338.218337] Lustre: Skipped 3 previous similar messages [11339.167493] Lustre: 330592:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787873269/real 1787873269] req@00000000f0759e07 x1874720812231424/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787873285 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11339.192431] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11339.207245] Lustre: Skipped 1 previous similar message [11339.231556] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11339.406290] Lustre: server umount lustre-MDT0000 complete [11342.670382] LustreError: 336014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873289 with bad export cookie 6277993999119510035 [11342.681989] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11342.688786] LustreError: 336014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [11343.021161] Lustre: server umount lustre-MDT0001 complete [11353.578445] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [11356.479839] Key type lgssc unregistered [11356.763783] LNet: 339471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11356.769159] LNetError: 339471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11356.807521] LNet: Removed LNI 192.168.203.109@tcp [11357.572822] Key type .llcrypt unregistered [11357.576296] Key type ._llcrypt unregistered [11375.217150] Key type ._llcrypt registered [11375.218490] Key type .llcrypt registered [11375.312862] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [11391.729329] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [11393.250286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11393.315085] alg: No test for adler32 (adler32-zlib) [11394.444276] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [11394.699290] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [11396.367175] Key type lgssc registered [11398.504978] Lustre: Echo OBD driver; http://www.lustre.org/ [11442.319652] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11442.353088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11443.764251] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11443.807832] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11443.891479] Lustre: lustre-MDT0000: new disk, initializing [11444.182821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11444.227592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11449.078213] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11459.072143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11459.182548] Lustre: 342634:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11459.240458] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11459.245781] Lustre: Skipped 1 previous similar message [11459.314484] Lustre: lustre-MDT0001: new disk, initializing [11459.417761] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11459.465522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11459.483639] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11464.408512] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11473.898882] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11479.839114] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11486.897835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11487.165868] Lustre: lustre-OST0000: new disk, initializing [11487.170632] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11487.194330] Lustre: 344600:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11487.296874] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11493.306166] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11496.557753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11496.571949] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11496.649606] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11502.633405] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11507.782523] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11508.004229] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11512.572877] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11512.820321] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11517.410856] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11517.424345] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11517.429532] LustreError: Skipped 1 previous similar message [11517.450662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11517.466100] Lustre: Skipped 1 previous similar message [11520.844156] Lustre: server umount lustre-OST0000 complete [11526.115089] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11526.121507] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11526.129786] Lustre: Skipped 1 previous similar message [11526.133832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11527.649045] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11527.653653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11527.677443] Lustre: Skipped 1 previous similar message [11530.911874] Lustre: server umount lustre-MDT0000 complete [11534.902438] LustreError: 342625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873481 with bad export cookie 4953182882311412753 [11534.905122] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11534.924305] LustreError: 342625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11535.231453] Lustre: server umount lustre-MDT0001 complete [11544.534521] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 19:31:29 (1787873489) [11560.218868] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [11570.489853] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [11604.187311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11604.498148] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11604.535219] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11604.640261] Lustre: lustre-MDT0000: new disk, initializing [11604.752487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11604.769532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11609.122091] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11619.698855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11619.837684] Lustre: 349362:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11619.910285] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11619.915427] Lustre: Skipped 1 previous similar message [11620.019312] Lustre: lustre-MDT0001: new disk, initializing [11620.145296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11620.195483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11620.212948] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11624.914729] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11634.591803] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11639.981945] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11646.247462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11646.588769] Lustre: lustre-OST0000: new disk, initializing [11646.595101] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11646.604525] Lustre: 351332:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11646.727903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11652.141977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11652.151345] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11652.283403] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11653.535513] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11662.371563] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11698.146132] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11698.152833] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11698.168773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11700.560533] Lustre: server umount lustre-OST0000 complete [11707.362933] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11707.368514] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11707.376699] Lustre: Skipped 1 previous similar message [11707.381139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11707.384892] Lustre: Skipped 1 previous similar message [11710.620761] Lustre: server umount lustre-MDT0000 complete [11714.021969] LustreError: 349369:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11714.021969] LustreError: 349374:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11714.021983] LustreError: 349374:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [11714.385870] LustreError: 349354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873661 with bad export cookie 4953182882311413642 [11714.396247] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11714.406413] LustreError: 349354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11714.845719] Lustre: server umount lustre-MDT0001 complete [11725.819380] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [11728.615961] Key type lgssc unregistered [11728.915918] LNet: 353507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11728.924781] LNetError: 353507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11728.959285] LNet: Removed LNI 192.168.203.109@tcp [11729.771771] Key type .llcrypt unregistered [11729.776368] Key type ._llcrypt unregistered [11817.746904] Key type ._llcrypt registered [11817.750966] Key type .llcrypt registered [11817.876364] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [11818.244473] Key type .llcrypt unregistered [11818.246620] Key type ._llcrypt unregistered [11891.375558] Key type ._llcrypt registered [11891.379203] Key type .llcrypt registered [11891.525234] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [11891.928494] Key type .llcrypt unregistered [11891.933413] Key type ._llcrypt unregistered [11923.221347] Key type ._llcrypt registered [11923.225363] Key type .llcrypt registered [11923.362284] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [11923.795528] Key type .llcrypt unregistered [11923.799032] Key type ._llcrypt unregistered [11950.979627] Key type ._llcrypt registered [11950.984524] Key type .llcrypt registered [11951.087469] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [11951.360154] Key type .llcrypt unregistered [11951.366949] Key type ._llcrypt unregistered [12021.400487] Key type ._llcrypt registered [12021.403426] Key type .llcrypt registered [12021.509550] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [12021.767415] Key type .llcrypt unregistered [12021.769806] Key type ._llcrypt unregistered [12052.501532] Key type ._llcrypt registered [12052.504778] Key type .llcrypt registered [12052.600508] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [12052.906509] Key type .llcrypt unregistered [12052.910745] Key type ._llcrypt unregistered [12080.001866] Key type ._llcrypt registered [12080.004487] Key type .llcrypt registered [12080.150784] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [12080.506974] Key type .llcrypt unregistered [12080.509954] Key type ._llcrypt unregistered [12109.664881] Key type ._llcrypt registered [12109.667585] Key type .llcrypt registered [12109.799847] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [12110.140313] Key type .llcrypt unregistered [12110.143327] Key type ._llcrypt unregistered [12138.107622] Key type ._llcrypt registered [12138.109829] Key type .llcrypt registered [12138.211621] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [12138.502353] Key type .llcrypt unregistered [12138.504492] Key type ._llcrypt unregistered [12166.629360] Key type ._llcrypt registered [12166.636581] Key type .llcrypt registered [12166.803751] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [12167.149374] Key type .llcrypt unregistered [12167.157308] Key type ._llcrypt unregistered [12222.001633] Key type ._llcrypt registered [12222.004412] Key type .llcrypt registered [12222.123435] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [12222.412265] Key type .llcrypt unregistered [12222.415301] Key type ._llcrypt unregistered [12249.103649] Key type ._llcrypt registered [12249.107187] Key type .llcrypt registered [12249.195297] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [12249.498323] Key type .llcrypt unregistered [12249.499809] Key type ._llcrypt unregistered [12276.018946] Key type ._llcrypt registered [12276.028513] Key type .llcrypt registered [12276.136105] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [12276.584426] Key type .llcrypt unregistered [12276.586978] Key type ._llcrypt unregistered [12327.166387] Key type ._llcrypt registered [12327.170069] Key type .llcrypt registered [12327.358901] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [12344.401411] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [12345.372472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12345.493687] alg: No test for adler32 (adler32-zlib) [12346.648320] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [12346.955547] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [12348.631158] Key type lgssc registered [12349.966607] Lustre: Echo OBD driver; http://www.lustre.org/ [12384.030955] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12384.055188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12385.309941] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12385.355284] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12385.450170] Lustre: tst250fs-MDT0000: new disk, initializing [12385.562301] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12385.582255] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12389.991371] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [12399.745144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12399.858772] Lustre: 373805:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12399.936001] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12399.939995] Lustre: Skipped 1 previous similar message [12400.011959] Lustre: tst250fs-MDT0001: new disk, initializing [12400.108377] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12400.154855] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12400.177388] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12404.058153] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [12408.105942] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12413.889319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12414.130580] Lustre: tst250fs-OST0000: new disk, initializing [12414.135797] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [12414.141798] Lustre: 375436:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12414.145251] Lustre: tst250fs-OST0000: Not available for connect from 0@lo (not set up) [12414.271715] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12419.626928] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12419.632614] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12419.656056] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [12420.514116] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [12429.366302] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12429.523197] Lustre: tst250fs-OST0001: new disk, initializing [12429.527236] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [12429.534187] Lustre: 376305:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12429.631035] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12431.081200] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12431.092236] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12431.203684] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12435.998741] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [12441.267306] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [12443.482815] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [12443.492095] Lustre: Skipped 2 previous similar messages [12461.541177] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12461.550417] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12461.567638] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12462.054778] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12462.090323] Lustre: Skipped 2 previous similar messages [12467.174414] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12467.188523] Lustre: Skipped 7 previous similar messages [12467.505791] Lustre: server umount tst250fs-MDT0000 complete [12470.952656] LustreError: 376316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787874417 with bad export cookie 18057654738906178294 [12470.954742] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12470.956868] LustreError: 376316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12471.209941] Lustre: server umount tst250fs-MDT0001 complete [12484.942295] Lustre: server umount tst250fs-OST0000 complete [12499.234802] Lustre: server umount tst250fs-OST0001 complete [12515.532833] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [12524.032146] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [12553.998039] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 19:48:19 (1787874499) [12555.674782] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [12570.359980] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [12580.945557] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [12612.168585] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12344 sec ========= 19:49:16 (1787874556) [12614.035889] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:49:18 (1787874558) === [12617.722966] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:49:22 (1787874562) === [12657.167403] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [12659.708723] Key type lgssc unregistered [12659.933315] LNet: 385437:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12659.938779] LNetError: 385437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12659.950919] LNet: Removed LNI 192.168.203.109@tcp [12660.696175] Key type .llcrypt unregistered [12660.697876] Key type ._llcrypt unregistered