[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 489753711 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003296] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.007960] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008017] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009021] pid_max: default: 32768 minimum: 301 [ 0.010125] LSM: Security Framework initializing [ 0.011053] Yama: becoming mindful. [ 0.012050] SELinux: Initializing. [ 0.013080] *** VALIDATE selinux *** [ 0.021705] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025716] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027046] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028095] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029099] *** VALIDATE tmpfs *** [ 0.031360] *** VALIDATE proc *** [ 0.032217] *** VALIDATE cgroup *** [ 0.033007] *** VALIDATE cgroup2 *** [ 0.034254] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035136] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037027] Spectre V2 : User space: Vulnerable [ 0.038006] Speculative Store Bypass: Vulnerable [ 0.040540] debug: unmapping init [mem 0xffffffff9c459000-0xffffffff9c460fff] [ 0.042168] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043624] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044018] ... version: 2 [ 0.045011] ... bit width: 48 [ 0.046012] ... generic registers: 4 [ 0.047013] ... value mask: 0000ffffffffffff [ 0.048012] ... max period: 00007fffffffffff [ 0.049018] ... fixed-purpose events: 3 [ 0.050007] ... event mask: 000000070000000f [ 0.051294] rcu: Hierarchical SRCU implementation. [ 0.053401] smp: Bringing up secondary CPUs ... [ 0.054667] x86: Booting SMP configuration: [ 0.055023] .... node #0, CPUs: #1 #2 #3 [ 0.058453] smp: Brought up 1 node, 4 CPUs [ 0.060011] smpboot: Max logical packages: 1 [ 0.061015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.139000] node 0 deferred pages initialised in 74ms [ 0.142173] devtmpfs: initialized [ 0.143241] x86/mm: Memory block size: 128MB [ 0.145920] gcov: version magic: 0x41383552 [ 0.146578] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.148086] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.151269] pinctrl core: initialized pinctrl subsystem [ 0.153153] [ 0.153832] ************************************************************* [ 0.157011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159011] ** ** [ 0.161010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.164012] ** ** [ 0.166007] ** This means that this kernel is built to expose internal ** [ 0.168013] ** IOMMU data structures, which may compromise security on ** [ 0.170012] ** your system. ** [ 0.172009] ** ** [ 0.175012] ** If you see this message and you are not debugging the ** [ 0.177008] ** kernel, report this immediately to your vendor! ** [ 0.179011] ** ** [ 0.182010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.184023] ************************************************************* [ 0.187666] NET: Registered protocol family 16 [ 0.189414] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.192059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.195057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.199088] cpuidle: using governor menu [ 0.201000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.203405] PCI: Using configuration type 1 for base access [ 0.205120] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.214113] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.216021] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.220175] cryptd: max_cpu_qlen set to 1000 [ 0.225010] ACPI: Added _OSI(Module Device) [ 0.227016] ACPI: Added _OSI(Processor Device) [ 0.229013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.230012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.236202] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.243084] ACPI: Interpreter enabled [ 0.245050] ACPI: PM: (supports S0 S3 S4 S5) [ 0.246026] ACPI: Using IOAPIC for interrupt routing [ 0.248184] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.252535] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.265506] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.267040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.270019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.274082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.279499] acpiphp: Slot [2] registered [ 0.281099] acpiphp: Slot [5] registered [ 0.284158] acpiphp: Slot [6] registered [ 0.286131] acpiphp: Slot [7] registered [ 0.288134] acpiphp: Slot [8] registered [ 0.290080] acpiphp: Slot [9] registered [ 0.292196] acpiphp: Slot [10] registered [ 0.294127] acpiphp: Slot [3] registered [ 0.295096] acpiphp: Slot [4] registered [ 0.297091] acpiphp: Slot [11] registered [ 0.298098] acpiphp: Slot [12] registered [ 0.300166] acpiphp: Slot [13] registered [ 0.301101] acpiphp: Slot [14] registered [ 0.302081] acpiphp: Slot [15] registered [ 0.303076] acpiphp: Slot [16] registered [ 0.305117] acpiphp: Slot [17] registered [ 0.307110] acpiphp: Slot [18] registered [ 0.308163] acpiphp: Slot [19] registered [ 0.310095] acpiphp: Slot [20] registered [ 0.311122] acpiphp: Slot [21] registered [ 0.314082] acpiphp: Slot [22] registered [ 0.316120] acpiphp: Slot [23] registered [ 0.318177] acpiphp: Slot [24] registered [ 0.320142] acpiphp: Slot [25] registered [ 0.321118] acpiphp: Slot [26] registered [ 0.323111] acpiphp: Slot [27] registered [ 0.325117] acpiphp: Slot [28] registered [ 0.327117] acpiphp: Slot [29] registered [ 0.329219] acpiphp: Slot [30] registered [ 0.332123] acpiphp: Slot [31] registered [ 0.334065] PCI host bridge to bus 0000:00 [ 0.337013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.339021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.340016] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.343092] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.346021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.349031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.351171] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.354978] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.358384] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.370013] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.374891] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.377030] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.380016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.382017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.385613] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.388833] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.391040] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.395386] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.401013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.413012] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.418014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.424088] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.433015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.440014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.455014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.465142] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.474012] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.485014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.512014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.525789] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.537015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.549015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.593024] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.612720] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.619018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.625015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.640015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.647644] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.653033] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.660019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.681016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.690300] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.701024] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.719023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.747021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.762656] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.765493] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.768887] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.771489] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.774221] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.778186] iommu: Default domain type: Passthrough [ 0.780539] SCSI subsystem initialized [ 0.781261] ACPI: bus type USB registered [ 0.784100] usbcore: registered new interface driver usbfs [ 0.786098] usbcore: registered new interface driver hub [ 0.789079] usbcore: registered new device driver usb [ 0.791183] pps_core: LinuxPPS API ver. 1 registered [ 0.793009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.796060] PTP clock support registered [ 0.798174] EDAC MC: Ver: 3.0.0 [ 0.800142] PCI: Using ACPI for IRQ routing [ 0.801483] NetLabel: Initializing [ 0.803014] NetLabel: domain hash size = 128 [ 0.804010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.806134] NetLabel: unlabeled traffic allowed by default [ 0.809139] vgaarb: loaded [ 0.810346] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.812021] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.819328] clocksource: Switched to clocksource kvm-clock [ 0.927829] VFS: Disk quotas dquot_6.6.0 [ 0.929853] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.933213] *** VALIDATE ramfs *** [ 0.934795] *** VALIDATE hugetlbfs *** [ 0.936822] pnp: PnP ACPI init [ 0.939833] pnp: PnP ACPI: found 6 devices [ 0.960142] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.964560] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.967061] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.968761] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.970816] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.973030] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.976289] NET: Registered protocol family 2 [ 0.979106] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.984249] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.987771] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.992713] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.996563] TCP: Hash tables configured (established 65536 bind 65536) [ 0.999357] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.002618] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.006119] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.009494] NET: Registered protocol family 1 [ 1.011910] RPC: Registered named UNIX socket transport module. [ 1.014208] RPC: Registered udp transport module. [ 1.016467] RPC: Registered tcp transport module. [ 1.018036] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.020693] NET: Registered protocol family 44 [ 1.022557] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.025190] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.028229] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.030964] PCI: CLS 0 bytes, default 64 [ 1.033213] Unpacking initramfs... [ 2.454713] debug: unmapping init [mem 0xffff99c53cc54000-0xffff99c53ffbffff] [ 2.458867] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.461109] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.464436] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.970870] Initialise system trusted keyrings [ 2.972887] Key type blacklist registered [ 2.980107] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.989901] zbud: loaded [ 2.993394] *** VALIDATE nfs *** [ 2.994821] *** VALIDATE nfs4 *** [ 2.996899] pstore: using deflate compression [ 3.004831] Platform Keyring initialized [ 3.130491] NET: Registered protocol family 38 [ 3.132526] Key type asymmetric registered [ 3.134112] Asymmetric key parser 'x509' registered [ 3.136089] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.139585] io scheduler mq-deadline registered [ 3.141293] io scheduler kyber registered [ 3.142789] io scheduler bfq registered [ 3.145167] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.147853] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.150461] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.153228] ACPI: Power Button [PWRF] [ 3.158697] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.166473] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.189366] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.198179] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.221467] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.249550] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.277079] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.283923] Non-volatile memory driver v1.3 [ 3.285474] Linux agpgart interface v0.103 [ 3.324402] virtio_blk virtio1: [vda] 146624 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.327744] vda: detected capacity change from 0 to 75071488 [ 3.348452] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.351319] vdb: detected capacity change from 0 to 1073741824 [ 3.366809] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.371305] vdc: detected capacity change from 0 to 2621440000 [ 3.391244] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.394222] vdd: detected capacity change from 0 to 2621440000 [ 3.409186] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.412481] vde: detected capacity change from 0 to 4294967296 [ 3.427531] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.431013] vdf: detected capacity change from 0 to 4294967296 [ 3.437812] libphy: Fixed MDIO Bus: probed [ 3.442802] usbcore: registered new interface driver usbserial_generic [ 3.444762] usbserial: USB Serial support registered for generic [ 3.446756] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.451344] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.453311] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.455705] mousedev: PS/2 mouse device common for all mice [ 3.458598] rtc_cmos 00:05: RTC can wake from S4 [ 3.460738] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.465239] rtc_cmos 00:05: registered as rtc0 [ 3.467753] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.468320] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.475137] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.475166] intel_pstate: CPU model not supported [ 3.481566] hid: raw HID events driver (C) Jiri Kosina [ 3.484093] usbcore: registered new interface driver usbhid [ 3.486421] usbhid: USB HID core driver [ 3.488356] drop_monitor: Initializing network drop monitor service [ 3.491192] Initializing XFRM netlink socket [ 3.493237] NET: Registered protocol family 10 [ 3.496548] Segment Routing with IPv6 [ 3.498374] NET: Registered protocol family 17 [ 3.500541] mpls_gso: MPLS GSO support [ 3.506838] RAS: Correctable Errors collector initialized. [ 3.509176] AVX version of gcm_enc/dec engaged. [ 3.511186] AES CTR mode by8 optimization enabled [ 3.586386] sched_clock: Marking stable (3586356761, 0)->(4496170399, -909813638) [ 3.591510] registered taskstats version 1 [ 3.593474] Loading compiled-in X.509 certificates [ 3.595590] zswap: loaded using pool lzo/zbud [ 3.621918] Key type big_key registered [ 3.635408] Key type encrypted registered [ 3.637013] ima: No TPM chip found, activating TPM-bypass! [ 3.639148] ima: Allocated hash algorithm: sha1 [ 3.640678] ima: No architecture policies found [ 3.642499] evm: Initialising EVM extended attributes: [ 3.644414] evm: security.selinux [ 3.645734] evm: security.ima [ 3.646751] evm: security.capability [ 3.647899] evm: HMAC attrs: 0x1 [ 3.650472] rtc_cmos 00:05: setting system clock to 2026-08-27 11:29:08 UTC (1787830148) [ 3.657094] debug: unmapping init [mem 0xffffffff9d403000-0xffffffff9d5fffff] [ 3.660314] debug: unmapping init [mem 0xffffffff9c182000-0xffffffff9c458fff] [ 3.670145] Write protecting the kernel read-only data: 28672k [ 3.675177] debug: unmapping init [mem 0xffffffff9a803000-0xffffffff9a9fffff] [ 3.677627] debug: unmapping init [mem 0xffffffff9b114000-0xffffffff9b1fffff] [ 3.709472] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.717903] systemd[1]: Detected virtualization kvm. [ 3.719876] systemd[1]: Detected architecture x86-64. [ 3.722116] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.750551] systemd[1]: No hostname configured. [ 3.752752] systemd[1]: Set hostname to . [ 3.755290] random: systemd: uninitialized urandom read (16 bytes read) [ 3.757944] systemd[1]: Initializing machine ID from random generator. [ 3.802446] random: ln: uninitialized urandom read (6 bytes read) [ 3.897517] random: systemd: uninitialized urandom read (16 bytes read) [ 3.900548] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.905825] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.911561] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.556839] device-mapper: uevent: version 1.0.3 [ 4.559936] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.306432] random: fast init done [ 5.317962] virtio_net virtio0 ens2: renamed from eth0 [ 5.406071] scsi host0: ata_piix [ 5.444275] scsi host1: ata_piix [ 5.446207] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.448903] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.800506] dracut-initqueue[593]: RTNETLINK answers: File exists [ 10.156841] random: crng init done [ 10.158557] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.612348] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.758353] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.016983] SELinux: Disabled at runtime. [ 12.078175] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.087482] systemd[1]: Detected virtualization kvm. [ 12.090163] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.581439] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.584774] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.589238] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.593439] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.597722] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.604544] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.612381] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. [ 12.674401] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.061302] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.384367] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.438425] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.508104] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.526237] EDAC sbridge: Ver: 1.1.2 [ 15.208748] Key type dns_resolver registered [ 15.510459] NFS: Registering the id_resolver key type [ 15.512378] Key type id_resolver registered [ 15.514011] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg645-server login: [ 50.617147] libcfs: loading out-of-tree module taints kernel. [ 50.678429] Key type ._llcrypt registered [ 50.682548] Key type .llcrypt registered [ 50.842315] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 72.408602] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 74.351789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 74.373331] alg: No test for adler32 (adler32-zlib) [ 76.256889] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 76.940146] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 78.695183] Key type lgssc registered [ 80.749178] Lustre: Echo OBD driver; http://www.lustre.org/ [ 96.697406] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 110.479076] hrtimer: interrupt took 3702322 ns [ 137.452937] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 151.772661] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 151.815066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 153.142536] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 153.196378] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 153.369773] Lustre: lustre-MDT0000: new disk, initializing [ 153.504146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 153.540163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 158.926855] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 173.349393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 173.463330] Lustre: 6531:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 173.507788] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 173.514463] Lustre: Skipped 1 previous similar message [ 173.600731] Lustre: lustre-MDT0001: new disk, initializing [ 173.743109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 173.791377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 173.812184] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 179.069218] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 184.387703] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 194.051305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 194.351488] Lustre: lustre-OST0000: new disk, initializing [ 194.354509] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 194.366025] Lustre: 8468:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 194.426480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 198.694134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 198.708227] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 198.808309] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 201.327719] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 216.259364] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 216.517365] Lustre: lustre-OST0001: new disk, initializing [ 216.521051] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 216.528323] Lustre: 9543:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 216.641439] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 223.256968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 223.264321] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 223.406345] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 224.249362] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 235.585103] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 240.459689] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 247.099625] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing check_logdir /tmp/testlogs/ [ 252.588765] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing yml_node [ 257.895700] Lustre: DEBUG MARKER: Client: 2.17.57.61 [ 260.700709] Lustre: DEBUG MARKER: MDS: 2.17.57.61 [ 263.584646] Lustre: DEBUG MARKER: OSS: 2.17.57.61 [ 265.796296] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 07:33:27 EDT 2026 [ 285.353953] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 287.103942] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 305.127386] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 305.129382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 305.141417] Lustre: Skipped 3 previous similar messages [ 305.165425] Lustre: Skipped 3 previous similar messages [ 310.314437] Lustre: server umount lustre-MDT0000 complete [ 315.362291] LustreError: 6540:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 315.372063] LustreError: 6540:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 316.139305] LustreError: 6524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830460 with bad export cookie 174069370163661830 [ 316.145027] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 316.151315] LustreError: 6524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 320.485560] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 320.488345] LustreError: 6538:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 320.496622] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 320.529430] LustreError: 6538:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 320.550316] Lustre: Skipped 1 previous similar message [ 321.505031] LustreError: 9550:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 321.506275] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 321.537212] LustreError: 9550:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 321.546763] Lustre: Skipped 1 previous similar message [ 322.578750] Lustre: server umount lustre-MDT0001 complete [ 327.854374] Lustre: server umount lustre-OST0000 complete [ 332.079206] Lustre: server umount lustre-OST0001 complete [ 339.104640] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 347.676776] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 383.333504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 383.681183] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 383.723369] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 383.822634] Lustre: lustre-MDT0000: new disk, initializing [ 383.907470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 383.929475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 388.359826] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 399.955304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 400.132255] Lustre: 15100:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 400.217798] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 400.224055] Lustre: Skipped 1 previous similar message [ 400.381469] Lustre: lustre-MDT0001: new disk, initializing [ 400.556640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 400.592742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 400.611101] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 405.056604] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 414.888295] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 420.852466] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 427.600741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 427.827877] Lustre: lustre-OST0000: new disk, initializing [ 427.830975] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 427.844614] Lustre: 17071:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 427.937306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 429.897570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 429.907203] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 429.987828] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 433.765527] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 442.941650] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 447.387103] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 447.605977] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 452.251952] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 452.425415] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 455.651583] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 455.652179] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 455.658282] Lustre: Skipped 1 previous similar message [ 455.660553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 455.665699] LustreError: Skipped 1 previous similar message [ 455.697473] Lustre: Skipped 1 previous similar message [ 460.291177] Lustre: server umount lustre-OST0000 complete [ 465.888552] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 465.890737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 465.905081] Lustre: Skipped 2 previous similar messages [ 465.918017] Lustre: Skipped 1 previous similar message [ 470.380894] Lustre: server umount lustre-MDT0000 complete [ 471.010953] LustreError: 15979:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 471.038849] LustreError: 15979:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 474.312102] LustreError: 15092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830619 with bad export cookie 174069370163663363 [ 474.320224] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 474.322574] LustreError: 15092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 474.603178] Lustre: server umount lustre-MDT0001 complete [ 479.863293] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 07:37:02 (1787830622) [ 498.406557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 499.106162] LustreError: 19945:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 499.241526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 503.836528] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 508.384741] LustreError: 19946:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 508.406371] LustreError: 19946:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 513.538452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 518.474240] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 526.066277] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 531.727776] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 533.880241] Lustre: Failing over lustre-MDT0000 [ 534.226926] Lustre: server umount lustre-MDT0000 complete [ 534.500532] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 534.515845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 534.545673] LustreError: 19946:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 534.569808] LustreError: 19946:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 553.764985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 553.859732] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 558.825542] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 559.204298] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 559.208761] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 559.254256] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 567.470977] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 571.315368] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 571.329447] Lustre: Skipped 1 previous similar message [ 571.335233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 571.339183] Lustre: Skipped 1 previous similar message [ 577.965109] Lustre: server umount lustre-MDT0000 complete [ 581.601367] LustreError: 19940:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 581.624328] LustreError: 19940:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 583.076411] LustreError: 19927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830727 with bad export cookie 174069370163664777 [ 583.085939] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 583.093281] LustreError: 19927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 583.816796] Lustre: server umount lustre-MDT0001 complete [ 593.704348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 594.270686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 594.272929] Lustre: Skipped 2 previous similar messages [ 599.552241] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 611.257815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 617.360530] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 625.683119] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 631.334687] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 633.466034] Lustre: Failing over lustre-MDT0000 [ 633.686341] Lustre: server umount lustre-MDT0000 complete [ 636.580227] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 636.596046] Lustre: Skipped 2 previous similar messages [ 647.157107] LustreError: 23275:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 647.199665] LustreError: 23275:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 652.258303] Lustre: 3668:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787830781/real 1787830781] req@ffff99c5b46a1680 x1874675862004992/t0(0) o400->MGC192.168.206.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787830797 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 652.285573] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 653.010956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 663.522290] LustreError: 3665:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff99c589332580 x1874675862007936/t0(0) o250->MGC192.168.206.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 663.900615] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 669.167442] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 669.191312] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 669.199324] Lustre: Skipped 1 previous similar message [ 669.218652] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 669.301761] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 680.571483] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 686.562531] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 686.565822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 686.583017] Lustre: Skipped 1 previous similar message [ 686.594700] Lustre: Skipped 2 previous similar messages [ 691.956892] Lustre: server umount lustre-MDT0000 complete [ 696.068282] LustreError: 23261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830840 with bad export cookie 174069370163665715 [ 696.078323] LustreError: 23261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 696.089094] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 696.447253] Lustre: server umount lustre-MDT0001 complete [ 707.115255] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 07:40:48 (1787830848) [ 724.744478] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 733.471826] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 775.371661] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 785.778969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 785.986659] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 786.019857] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 786.097783] Lustre: lustre-MDT0000: new disk, initializing [ 786.166222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 786.171705] Lustre: Skipped 2 previous similar messages [ 786.186370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 790.642382] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 800.894374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 801.010670] Lustre: 29657:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 801.364631] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 805.325805] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 814.434690] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 820.685329] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 827.131981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 827.389829] Lustre: 31625:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 828.998649] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 829.044965] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 829.170986] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 829.187914] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 833.573572] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 843.340936] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 849.449295] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 870.370303] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 870.370885] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 870.390319] LustreError: Skipped 1 previous similar message [ 870.399797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 870.399807] Lustre: Skipped 3 previous similar messages [ 870.434908] Lustre: Skipped 1 previous similar message [ 884.191752] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 884.491345] Lustre: server umount lustre-OST0000 complete [ 895.021715] Lustre: server umount lustre-MDT0000 complete [ 895.968736] LustreError: 29664:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 895.988622] LustreError: 29664:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 898.381371] LustreError: 29648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831043 with bad export cookie 174069370163666121 [ 898.383454] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 898.393607] LustreError: 29648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 898.811677] Lustre: server umount lustre-MDT0001 complete [ 910.229698] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 913.278302] Key type lgssc unregistered [ 913.702978] LNet: 33611:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 913.708135] LNetError: 33611:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 913.731165] LNet: Removed LNI 192.168.206.145@tcp [ 914.726157] Key type .llcrypt unregistered [ 914.732216] Key type ._llcrypt unregistered [ 939.515592] Key type ._llcrypt registered [ 939.518521] Key type .llcrypt registered [ 939.594534] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 940.326827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 940.413178] alg: No test for adler32 (adler32-zlib) [ 941.494431] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 941.729888] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 943.383384] Key type lgssc registered [ 944.524547] Lustre: Echo OBD driver; http://www.lustre.org/ [ 951.993263] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 952.016431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 953.508543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 958.532501] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 964.064387] LustreError: 35040:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 964.098724] LustreError: 35040:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 968.214440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 968.577094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 973.047868] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 981.840325] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 988.195639] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 995.147642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 995.513890] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 998.665366] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1001.005198] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1008.620971] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1014.001549] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1027.781785] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 07:46:10 (1787831170) [ 1029.891294] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1031.892043] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1038.800621] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 07:46:21 (1787831181) [ 1047.723613] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 07:46:30 (1787831190) [ 1048.657968] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1056.737562] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1056.743319] Lustre: Skipped 2 previous similar messages [ 1058.580781] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 07:46:41 (1787831201) [ 1062.185062] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1062.190732] Lustre: Skipped 3 previous similar messages [ 1079.113966] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1079.122681] Lustre: Skipped 6 previous similar messages [ 1081.527266] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 07:47:04 (1787831224) [ 1084.062181] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1086.954562] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1094.497899] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 07:47:17 (1787831237) [ 1096.457744] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1096.466292] Lustre: Skipped 3 previous similar messages [ 1108.386499] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 07:47:31 (1787831251) [ 1126.880755] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1126.886898] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1126.893367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1126.909889] Lustre: Skipped 1 previous similar message [ 1126.922947] Lustre: Skipped 1 previous similar message [ 1129.768714] Lustre: server umount lustre-OST0000 complete [ 1137.121662] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1137.137359] Lustre: Skipped 1 previous similar message [ 1137.138379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1137.149144] Lustre: Skipped 1 previous similar message [ 1139.593392] Lustre: server umount lustre-MDT0000 complete [ 1142.240618] LustreError: 35768:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1142.273778] LustreError: 35768:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1143.450743] LustreError: 39103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831288 with bad export cookie 2988822178922297344 [ 1143.464869] LustreError: 39103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1143.465268] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1143.800261] Lustre: server umount lustre-MDT0001 complete [ 1154.574603] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 1157.160128] Key type lgssc unregistered [ 1157.471546] LNet: 42901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1157.485603] LNetError: 42901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1157.524651] LNet: Removed LNI 192.168.206.145@tcp [ 1158.397799] Key type .llcrypt unregistered [ 1158.399977] Key type ._llcrypt unregistered [ 1175.166949] Key type ._llcrypt registered [ 1175.169190] Key type .llcrypt registered [ 1175.254300] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 1189.458759] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 1190.553118] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1190.654046] alg: No test for adler32 (adler32-zlib) [ 1191.796338] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 1192.026733] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 1193.672057] Key type lgssc registered [ 1194.808105] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1231.612492] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1231.649126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1232.918159] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1232.937910] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1233.053465] Lustre: lustre-MDT0000: new disk, initializing [ 1233.156818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1233.204228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1237.950787] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1249.327678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1249.453046] Lustre: 46073:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1249.519215] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1249.522826] Lustre: Skipped 1 previous similar message [ 1249.625788] Lustre: lustre-MDT0001: new disk, initializing [ 1249.711269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1249.735884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1249.748633] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1253.846295] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1262.466509] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1267.485450] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1274.155231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1274.468502] Lustre: lustre-OST0000: new disk, initializing [ 1274.475213] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1274.486471] Lustre: 48041:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1274.561402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1281.102691] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1282.123614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1282.133366] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1282.221720] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1289.370296] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1293.382838] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1293.525601] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1297.556687] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1297.735030] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1303.020790] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1303.023206] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1303.029696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1303.037629] Lustre: Skipped 1 previous similar message [ 1305.707552] Lustre: server umount lustre-OST0000 complete [ 1311.200319] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1311.204386] LustreError: Skipped 1 previous similar message [ 1311.207531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1311.216980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1311.220636] Lustre: Skipped 1 previous similar message [ 1313.248744] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1313.249049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1313.269852] Lustre: Skipped 1 previous similar message [ 1314.825622] Lustre: server umount lustre-MDT0000 complete [ 1318.148884] LustreError: 49583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831462 with bad export cookie 14029231870138878914 [ 1318.161361] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1318.169508] LustreError: 49583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1318.635359] Lustre: server umount lustre-MDT0001 complete [ 1327.121255] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 07:51:10 (1787831470) [ 1332.880585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1333.250328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1337.878535] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1338.341461] LustreError: 50501:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1338.361367] LustreError: 50501:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1343.459249] LustreError: 50502:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1346.546302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1346.989565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1351.245792] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1358.167763] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1363.960316] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1370.594612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1370.930431] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1377.646019] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1384.809407] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1389.043732] LustreError: 53029:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1389.952920] Lustre: Modifying parameter general.timeout=1 in log params [ 1391.549588] Lustre: Modifying parameter general.timeout=3 in log params [ 1391.552554] Lustre: Skipped 1 previous similar message [ 1394.289647] Lustre: Modifying parameter general.timeout=6 in log params [ 1394.293457] Lustre: Skipped 2 previous similar messages [ 1398.603412] Lustre: Modifying parameter general.timeout=11 in log params [ 1398.607094] Lustre: Skipped 4 previous similar messages [ 1407.161162] Lustre: Modifying parameter general.timeout=22 in log params [ 1407.166409] Lustre: Skipped 10 previous similar messages [ 1423.903923] Lustre: Modifying parameter general.timeout=42 in log params [ 1423.907812] Lustre: Skipped 19 previous similar messages [ 1456.618183] Lustre: Modifying parameter general.timeout=80 in log params [ 1456.625087] Lustre: Skipped 37 previous similar messages [ 1511.428713] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 07:54:14 (1787831654) [ 1513.559789] Lustre: 59968:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1514.936196] Lustre: 60066:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1517.880698] Lustre: 60280:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1517.892691] Lustre: 60280:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1522.857058] Lustre: Disabling parameter general.ost.OSS.ost_io.nrs_tbf_rule= in log params [ 1522.861442] Lustre: Skipped 57 previous similar messages [ 1545.038941] Lustre: 61831:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1545.043906] Lustre: 61831:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1545.070133] LustreError: 61831:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1549.118442] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 07:54:51 (1787831691) [ 1562.080090] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1562.084125] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1562.093642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1565.999609] Lustre: server umount lustre-MDT0000 complete [ 1568.943155] LustreError: 50481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831713 with bad export cookie 14029231870138879796 [ 1568.946488] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1568.951996] LustreError: 50481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1569.209922] Lustre: server umount lustre-MDT0001 complete [ 1582.862535] Lustre: server umount lustre-OST0000 complete [ 1607.418482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1607.829815] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1608.093779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1612.621984] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1621.464758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1621.554094] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1621.575589] Lustre: 64363:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1626.084607] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1632.798757] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1638.036250] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1644.747103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1645.050362] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1645.372638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1645.384579] Lustre: Skipped 1 previous similar message [ 1651.856794] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1658.326678] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1659.875731] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1659.892386] Lustre: Skipped 15 previous similar messages [ 1659.923477] Lustre: 66929:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1669.600597] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1669.609325] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1669.616113] LustreError: Skipped 1 previous similar message [ 1669.618687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1669.618696] Lustre: Skipped 3 previous similar messages [ 1669.661590] Lustre: Skipped 3 previous similar messages [ 1674.466860] Lustre: server umount lustre-OST0000 complete [ 1679.847405] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1679.850215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1679.865832] Lustre: Skipped 1 previous similar message [ 1679.870563] Lustre: Skipped 1 previous similar message [ 1684.860897] Lustre: server umount lustre-MDT0000 complete [ 1684.963192] LustreError: 64375:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1684.975493] LustreError: 64375:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1688.730391] LustreError: 64354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831833 with bad export cookie 14029231870138882764 [ 1688.732195] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1688.736577] LustreError: 64354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1689.140943] Lustre: server umount lustre-MDT0001 complete [ 1700.578193] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 1703.637325] Key type lgssc unregistered [ 1704.043663] LNet: 68145:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1704.062212] LNetError: 68145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1704.085586] LNet: Removed LNI 192.168.206.145@tcp [ 1705.332121] Key type .llcrypt unregistered [ 1705.334423] Key type ._llcrypt unregistered [ 1731.256482] Key type ._llcrypt registered [ 1731.262116] Key type .llcrypt registered [ 1731.345864] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 1732.350407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1732.489932] alg: No test for adler32 (adler32-zlib) [ 1733.642784] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 1733.908121] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 1735.599181] Key type lgssc registered [ 1736.458816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1743.889135] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1743.930814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1745.502028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1749.942711] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1755.622847] LustreError: 69561:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1755.635375] LustreError: 69561:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1758.412367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1759.038741] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1763.672954] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1771.327319] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1776.772772] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1783.321500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1783.829016] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1790.337230] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1797.065418] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1803.651531] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1811.430271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1811.433029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1811.442801] Lustre: Skipped 2 previous similar messages [ 1811.450593] Lustre: Skipped 2 previous similar messages [ 1816.545860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1816.550283] Lustre: Skipped 1 previous similar message [ 1817.107532] Lustre: server umount lustre-MDT0000 complete [ 1820.396507] LustreError: 69542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831965 with bad export cookie 465505883869704567 [ 1820.405618] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1820.415525] LustreError: 69542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1820.659822] Lustre: server umount lustre-MDT0001 complete [ 1834.657186] Lustre: server umount lustre-OST0000 complete [ 1859.420858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1859.663952] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1859.734942] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1860.062500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1864.703637] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1873.112559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1873.212091] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1873.246219] Lustre: 74428:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1873.682074] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1877.293806] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1884.196212] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1890.062367] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1896.089456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1896.343839] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1896.694788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1902.309297] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1909.460680] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1910.829488] Lustre: 77005:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1912.514299] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1918.944802] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1918.948576] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1918.959974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1918.967833] Lustre: Skipped 1 previous similar message [ 1919.457477] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1924.591332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1924.602442] Lustre: Skipped 3 previous similar messages [ 1929.699691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1929.704637] Lustre: Skipped 1 previous similar message [ 1931.231492] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1931.383438] Lustre: server umount lustre-OST0000 complete [ 1934.816267] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1934.825737] Lustre: Skipped 2 previous similar messages [ 1939.938668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1939.949859] Lustre: Skipped 3 previous similar messages [ 1940.953908] Lustre: server umount lustre-MDT0000 complete [ 1943.855618] LustreError: 77008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832088 with bad export cookie 465505883869705400 [ 1943.858180] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1943.871715] LustreError: 77008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 1944.213357] Lustre: server umount lustre-MDT0001 complete [ 1953.814428] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 1956.552754] Key type lgssc unregistered [ 1956.809764] LNet: 78233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1956.821698] LNetError: 78233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1956.848482] LNet: Removed LNI 192.168.206.145@tcp [ 1957.949714] Key type .llcrypt unregistered [ 1957.951956] Key type ._llcrypt unregistered [ 1982.497213] Key type ._llcrypt registered [ 1982.504167] Key type .llcrypt registered [ 1982.614809] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 1983.590307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1983.639932] alg: No test for adler32 (adler32-zlib) [ 1984.685968] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 1984.912469] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 1986.551167] Key type lgssc registered [ 1987.449692] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1994.313261] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1994.332672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1995.772241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2000.075597] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2005.984691] LustreError: 79648:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2006.017509] LustreError: 79648:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2008.075070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2008.638022] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2013.245885] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2021.645402] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2027.031173] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2033.735312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2034.295782] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2040.224287] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2046.531933] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2052.160147] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2053.078777] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2054.700247] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2054.705866] Lustre: Skipped 1 previous similar message [ 2057.241703] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2057.252631] Lustre: Skipped 2 previous similar messages [ 2061.257292] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2061.266122] Lustre: Skipped 4 previous similar messages [ 2069.657143] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2069.664165] Lustre: Skipped 9 previous similar messages [ 2085.768615] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2085.772715] Lustre: Skipped 19 previous similar messages [ 2118.394113] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2118.402988] Lustre: Skipped 39 previous similar messages [ 2145.813995] Lustre: 87745:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2147.941653] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 08:04:50 (1787832290) [ 2149.391545] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2150.768628] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 08:04:53 (1787832293) [ 2157.539062] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2157.544165] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2157.563330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2158.560584] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2158.563514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2160.700864] Lustre: server umount lustre-OST0000 complete [ 2167.265814] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2167.279083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2167.298989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2167.308748] Lustre: Skipped 1 previous similar message [ 2171.605656] Lustre: server umount lustre-MDT0000 complete [ 2173.921435] LustreError: 79647:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2175.665592] LustreError: 80388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832320 with bad export cookie 7665167988736126194 [ 2175.673263] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2175.677440] LustreError: 80388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2176.156339] Lustre: server umount lustre-MDT0001 complete [ 2186.848537] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 2189.602301] Key type lgssc unregistered [ 2189.971332] LNet: 88997:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2190.015239] LNetError: 88997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2191.085192] LNet: Removed LNI 192.168.206.145@tcp [ 2192.031416] Key type .llcrypt unregistered [ 2192.033100] Key type ._llcrypt unregistered [ 2197.481793] Key type ._llcrypt registered [ 2197.485405] Key type .llcrypt registered [ 2197.706451] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2202.859921] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules [ 2203.890314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2203.950222] alg: No test for adler32 (adler32-zlib) [ 2205.381741] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 2205.823940] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 2207.703159] Key type lgssc registered [ 2209.260915] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2211.328524] LustreError: 90221:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2226.423951] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 2232.653862] LustreError: 90221:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2237.920404] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2237.974491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2239.957684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2245.508871] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2249.188927] LustreError: 91222:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2249.210524] LustreError: 91222:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2254.321834] LustreError: 91223:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2255.177397] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 08:06:37 (1787832397) [ 2259.437310] LustreError: 91222:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2259.995632] LustreError: 91255:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2260.017364] LustreError: 91255:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2260.031896] LustreError: 91255:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 2260.590783] Lustre: server umount lustre-MDT0000 complete [ 2274.810791] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 2277.813341] Key type lgssc unregistered [ 2278.162075] LNet: 92752:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2278.170176] LNetError: 92752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2278.191781] LNet: Removed LNI 192.168.206.145@tcp [ 2279.213500] Key type .llcrypt unregistered [ 2279.216362] Key type ._llcrypt unregistered [ 2295.889169] Key type ._llcrypt registered [ 2295.890793] Key type .llcrypt registered [ 2296.073342] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 2297.493823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2297.533711] alg: No test for adler32 (adler32-zlib) [ 2298.655923] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 2298.965067] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 2300.639194] Key type lgssc registered [ 2302.262465] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2311.799436] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2311.826300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2313.530932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2318.965393] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2323.952646] LustreError: 93971:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2323.987878] LustreError: 93971:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2329.074777] LustreError: 93970:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2330.290169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2331.312294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2336.000460] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2343.828667] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2349.601777] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2355.933396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2356.716137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2362.527143] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2370.803457] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2431.947699] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 08:09:33 (1787832573) [ 2433.591294] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2435.432273] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 08:09:37 (1787832577) [ 2442.729175] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2442.733497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2442.747905] Lustre: Skipped 2 previous similar messages [ 2442.758459] Lustre: Skipped 1 previous similar message [ 2445.895839] Lustre: server umount lustre-MDT0000 complete [ 2447.843492] LustreError: 93970:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2447.873161] LustreError: 93970:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2449.490275] LustreError: 93952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832594 with bad export cookie 231906636326382043 [ 2449.492061] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2449.507062] LustreError: 93952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2449.924048] Lustre: server umount lustre-MDT0001 complete [ 2464.998185] Lustre: server umount lustre-OST0000 complete [ 2476.814154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2477.224726] LustreError: 98273:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2477.377066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2482.661010] LustreError: 98274:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2482.763733] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2492.565216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2492.899760] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2492.912029] Lustre: Skipped 1 previous similar message [ 2493.275834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2498.735112] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2506.061323] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2511.963712] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2524.625879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2524.866907] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2524.873700] LustreError: 98263:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2524.891646] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2524.897781] LustreError: 100357:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2524.902513] LustreError: 100357:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2524.909467] LustreError: 100357:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2524.995596] Lustre: server umount lustre-OST0000 complete [ 2524.999907] LustreError: 100357:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2532.445746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2532.563809] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2532.571447] LustreError: 98263:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2532.578461] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2532.584602] LustreError: 100818:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2532.589542] LustreError: 100818:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2532.598186] LustreError: 100818:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2532.651915] Lustre: server umount lustre-OST0000 complete [ 2532.657607] LustreError: 100818:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2542.239588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2542.491564] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2542.497216] Lustre: Found index 0 for lustre-OST0000, updating log [ 2542.507093] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2542.637118] Lustre: lustre-OST0000: new disk, initializing [ 2542.642385] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2542.649152] Lustre: 101354:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2543.025477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2549.689554] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2553.176985] Lustre: Failing over lustre-OST0000 [ 2553.337716] Lustre: server umount lustre-OST0000 complete [ 2558.433664] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2558.435098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2558.438036] Lustre: Skipped 1 previous similar message [ 2558.443101] Lustre: Skipped 1 previous similar message [ 2562.531823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2562.541631] Lustre: Skipped 1 previous similar message [ 2566.993704] Lustre: server umount lustre-MDT0000 complete [ 2567.679915] LustreError: 98268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2567.712639] LustreError: 98268:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2572.943879] LustreError: 98253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832717 with bad export cookie 231906636326384787 [ 2572.966716] LustreError: 98253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2572.968207] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2573.775081] Lustre: server umount lustre-MDT0001 complete [ 2583.981241] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 08:12:06 (1787832726) [ 2600.192836] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 2609.771604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2610.202407] LustreError: 103698:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2610.218314] LustreError: 103698:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2610.348222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2614.467103] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2622.107380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2626.165274] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2628.803770] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2634.148721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2634.827825] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2635.814899] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2635.844423] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2635.846584] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2635.853171] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2640.499894] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2649.106138] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2649.227078] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2649.350321] Lustre: lustre-OST0001: new disk, initializing [ 2649.356909] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2649.361369] Lustre: 106016:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2649.754989] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2649.763735] Lustre: Skipped 2 previous similar messages [ 2656.335508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2656.349610] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2656.441052] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2656.550956] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2664.717329] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2671.035525] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2677.227890] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2677.231610] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2677.250417] Lustre: Skipped 2 previous similar messages [ 2677.253637] Lustre: Skipped 1 previous similar message [ 2682.376295] LustreError: 103694:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2682.390278] LustreError: 103694:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2682.434882] Lustre: server umount lustre-MDT0001 complete [ 2692.532978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2692.690707] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2692.699560] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2692.799253] Lustre: 103688:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2694.150815] LustreError: 107552:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2694.165577] LustreError: 107552:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2694.171897] LustreError: 103691:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2694.275279] LustreError: 107558:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.145@tcp: cfg command failed: rc = -17 [ 2694.302307] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2694.302307] [ 2698.152530] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2698.753343] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2707.777727] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 08:14:10 (1787832850) [ 2724.119211] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 2740.794530] Lustre: 109630:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2762.231704] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2765.337715] Lustre: 110767:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2846.687893] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2846.707413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2846.726755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2846.732411] Lustre: Skipped 3 previous similar messages [ 2852.506250] Lustre: server umount lustre-MDT0000 complete [ 2856.162491] LustreError: 103679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833001 with bad export cookie 231906636326385718 [ 2856.164829] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2856.175530] LustreError: 103679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2856.450565] Lustre: server umount lustre-MDT0001 complete [ 2871.007519] Lustre: server umount lustre-OST0000 complete [ 2873.823149] Lustre: 93284:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787833002/real 1787833002] req@ffff99c48cff8000 x1874678192104192/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787833018 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2873.855034] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2873.881813] Lustre: Skipped 3 previous similar messages [ 2875.103434] Lustre: server umount lustre-OST0001 complete [ 2880.358882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2889.491317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2898.106257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2907.702962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2925.364962] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 2936.004302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2936.025892] Lustre: lustre-MDT0000: reset Object Index mappings [ 2936.453453] LustreError: 114718:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2936.482017] LustreError: 114718:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 2936.634983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2936.643734] Lustre: Skipped 1 previous similar message [ 2941.060887] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2950.049934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2950.077268] Lustre: lustre-MDT0001: reset Object Index mappings [ 2956.930722] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2960.860387] Lustre: 115865:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2969.165396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2975.857575] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 2977.152680] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2982.399311] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 2985.962807] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2992.126961] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 2992.133406] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:225) [ 2993.089751] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 3001.070700] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3004.910823] Lustre: 117744:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3052.279187] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 08:19:54 (1787833194) [ 3079.139527] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3079.151296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3079.159669] Lustre: Skipped 2 previous similar messages [ 3079.179651] Lustre: Skipped 11 previous similar messages [ 3085.334972] Lustre: server umount lustre-MDT0000 complete [ 3088.620286] LustreError: 114701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833233 with bad export cookie 231906636326432793 [ 3088.636553] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3089.376185] LustreError: 114714:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3089.399803] LustreError: 114714:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 3095.198508] Lustre: server umount lustre-MDT0001 complete [ 3105.389983] Lustre: server umount lustre-OST0000 complete [ 3115.379616] Lustre: server umount lustre-OST0001 complete [ 3121.682721] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 3130.581708] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 3166.344042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3166.682305] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3166.725712] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3166.893391] Lustre: lustre-MDT0000: new disk, initializing [ 3166.973546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3166.990104] Lustre: Skipped 3 previous similar messages [ 3167.020255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3171.791457] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 3179.976840] Lustre: server umount lustre-MDT0000 complete [ 3188.793643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3189.061275] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3189.076441] Lustre: 122020:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3189.099984] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3193.520340] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 3198.193556] Lustre: server umount lustre-MDT0000 complete [ 3206.373847] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 08:22:29 (1787833349) [ 3207.990860] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3209.433438] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 08:22:32 (1787833352) [ 3226.349290] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 3235.223961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3239.518182] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 3247.523135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3247.645670] Lustre: 124068:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3247.700850] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3247.705062] Lustre: Skipped 1 previous similar message [ 3247.765229] Lustre: lustre-MDT0001: new disk, initializing [ 3247.867915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3247.876955] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3252.591130] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 3257.343421] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3263.169625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3263.372329] Lustre: lustre-OST0000: new disk, initializing [ 3263.375786] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3263.381145] Lustre: 125646:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3265.079676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3265.093803] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3265.206377] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3269.110732] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 3278.778145] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3278.949785] Lustre: lustre-OST0001: new disk, initializing [ 3278.952755] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3278.957361] Lustre: 126518:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3280.373328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3280.385841] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3280.490496] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3285.378222] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 3294.760368] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3298.233626] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3656.372339] Lustre: 132924:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3656.384193] Lustre: 132924:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3656.390522] Lustre: 132924:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3656.399240] Lustre: 132924:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3656.409739] Lustre: 132924:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3656.416560] Lustre: 132924:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3943.681533] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 08:34:45 (1787834085) [ 3959.776818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3959.789732] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3959.806124] Lustre: Skipped 3 previous similar messages [ 3959.809482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3959.812786] Lustre: Skipped 8 previous similar messages [ 3964.854508] Lustre: server umount lustre-MDT0000 complete [ 3966.437226] LustreError: 127334:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3966.452814] LustreError: 127334:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 3968.693906] LustreError: 132938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834113 with bad export cookie 231906636326500266 [ 3968.695262] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3968.702290] LustreError: 132938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3971.565186] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3971.568204] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3971.587973] Lustre: Skipped 4 previous similar messages [ 3971.605566] Lustre: Skipped 5 previous similar messages [ 3983.328358] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3983.687325] Lustre: server umount lustre-MDT0001 complete [ 3993.392879] Lustre: server umount lustre-OST0000 complete [ 4003.375530] Lustre: server umount lustre-OST0001 complete [ 4009.880255] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 4018.653712] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 4052.758677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4052.916732] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4052.940182] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4053.010907] Lustre: lustre-MDT0000: new disk, initializing [ 4053.062355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4053.071653] Lustre: Skipped 5 previous similar messages [ 4053.094924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4058.218613] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4068.339313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4068.460119] Lustre: 135992:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4068.514242] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4068.518163] Lustre: Skipped 1 previous similar message [ 4068.582397] Lustre: lustre-MDT0001: new disk, initializing [ 4068.701140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4068.711897] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4072.823488] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4082.243655] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4088.263423] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4094.687488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4094.893267] Lustre: lustre-OST0000: new disk, initializing [ 4094.896362] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4094.901254] Lustre: 137963:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4094.976381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4094.984419] Lustre: Skipped 1 previous similar message [ 4096.445620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4096.459640] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4096.580143] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4100.912558] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4109.940926] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4119.207879] Lustre: lustre-MDD0000: changelog on [ 4121.690290] Lustre: lustre-MDD0001: changelog on [ 6013.447317] Lustre: lustre-MDD0001: changelog off [ 6015.866497] Lustre: 139424:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6017.172033] Lustre: lustre-MDD0000: changelog off [ 6024.695491] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6024.698441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6024.716882] Lustre: Skipped 1 previous similar message [ 6024.726286] Lustre: Skipped 5 previous similar messages [ 6029.804977] LustreError: 139589:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6029.824859] LustreError: 139589:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 6030.009424] Lustre: server umount lustre-OST0000 complete [ 6034.919203] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6034.921271] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6034.933579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6034.940397] Lustre: Skipped 1 previous similar message [ 6040.033313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6040.035528] Lustre: Skipped 1 previous similar message [ 6041.034927] Lustre: server umount lustre-MDT0000 complete [ 6045.135138] LustreError: 135984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836189 with bad export cookie 231906636327317579 [ 6045.139461] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6045.152205] LustreError: 135984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6045.589402] Lustre: server umount lustre-MDT0001 complete [ 6059.393320] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 6062.595730] Key type lgssc unregistered [ 6062.841578] LNet: 147128:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6062.864597] LNetError: 147128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6062.895651] LNet: Removed LNI 192.168.206.145@tcp [ 6063.732224] Key type .llcrypt unregistered [ 6063.737182] Key type ._llcrypt unregistered [ 6085.670371] Key type ._llcrypt registered [ 6085.673474] Key type .llcrypt registered [ 6085.781621] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 6101.282838] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 6102.239063] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6102.308948] alg: No test for adler32 (adler32-zlib) [ 6103.409506] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 6103.699870] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 6105.384072] Key type lgssc registered [ 6106.452720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6135.715603] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6135.737536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6137.097111] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6137.134515] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6137.207272] Lustre: lustre-MDT0000: new disk, initializing [ 6137.315507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6137.340309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6141.191158] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6151.140893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6151.292480] Lustre: 150329:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6151.367135] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6151.371749] Lustre: Skipped 1 previous similar message [ 6151.450220] Lustre: lustre-MDT0001: new disk, initializing [ 6151.520703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6151.555549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6151.568058] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6155.754986] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6164.436573] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6169.592517] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6175.625732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6175.864426] Lustre: lustre-OST0000: new disk, initializing [ 6175.873562] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6175.882972] Lustre: 152298:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6175.944765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6181.701605] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6182.962059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6182.969952] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6183.101334] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6190.041770] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6195.483264] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_module obdecho/obdecho [ 6197.546052] LustreError: 153300:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6197.552822] LustreError: 153300:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6199.183854] LustreError: 153396:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6213.090354] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6213.113376] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6213.130299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6214.124829] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6214.142056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6214.153700] Lustre: Skipped 1 previous similar message [ 6214.165364] Lustre: Skipped 2 previous similar messages [ 6216.934745] Lustre: server umount lustre-MDT0000 complete [ 6221.094339] LustreError: 150320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836365 with bad export cookie 12141929017021733069 [ 6221.097824] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6221.114177] LustreError: 150320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6221.475463] Lustre: server umount lustre-MDT0001 complete [ 6235.542897] Lustre: server umount lustre-OST0000 complete [ 6264.847655] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 6277.441900] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 6317.519019] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6317.920716] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6317.972808] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6318.144090] Lustre: lustre-MDT0000: new disk, initializing [ 6318.276538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6318.310470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6323.080394] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6333.455933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6333.604754] Lustre: 157115:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6333.653650] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6333.657641] Lustre: Skipped 1 previous similar message [ 6333.731672] Lustre: lustre-MDT0001: new disk, initializing [ 6333.829937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6333.869987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6333.889858] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6338.612649] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6349.335412] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6356.134749] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6363.714875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6363.990401] Lustre: lustre-OST0000: new disk, initializing [ 6363.999354] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6364.008468] Lustre: 159083:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6364.109851] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6370.916564] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6373.441116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6373.454032] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6373.555723] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6380.637248] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6385.908734] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6386.120987] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6391.150976] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6391.438157] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6394.342979] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6394.347187] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6394.349417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6394.353295] Lustre: Skipped 1 previous similar message [ 6394.377563] LustreError: Skipped 1 previous similar message [ 6399.520385] Lustre: server umount lustre-OST0000 complete [ 6404.578344] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6404.589251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6404.596498] Lustre: Skipped 2 previous similar messages [ 6409.623996] Lustre: server umount lustre-MDT0000 complete [ 6409.702736] LustreError: 157125:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6409.718435] LustreError: 157125:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6413.055137] LustreError: 159082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836557 with bad export cookie 12141929017021734028 [ 6413.063409] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6413.070801] LustreError: 159082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6413.377790] Lustre: server umount lustre-MDT0001 complete [ 6416.665227] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 09:15:59 (1787836559) [ 6418.223931] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6419.815482] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 09:16:02 (1787836562) [ 6427.437307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6427.876027] LustreError: 161493:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6427.953709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6431.794908] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6433.254869] LustreError: 161494:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6437.346995] LustreError: 161493:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6439.533521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6443.435532] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6450.219116] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6455.983300] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6463.264149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6463.619854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6463.623962] Lustre: Skipped 1 previous similar message [ 6470.031201] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6477.593678] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6483.427916] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6483.434633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6483.439314] Lustre: Skipped 3 previous similar messages [ 6483.458112] Lustre: Skipped 2 previous similar messages [ 6483.719161] LustreError: 162759:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6483.815807] Lustre: server umount lustre-MDT0000 complete [ 6487.779159] LustreError: 161473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836632 with bad export cookie 12141929017021734910 [ 6487.791788] LustreError: 161473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6487.795711] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6492.578742] LustreError: 161489:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6492.580565] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6492.593184] LustreError: 161489:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6492.596162] Lustre: Skipped 2 previous similar messages [ 6502.367722] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6502.738684] Lustre: server umount lustre-MDT0001 complete [ 6509.630263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6521.819133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6522.002680] LustreError: 165333:0:(llog.c:1655:llog_backup()) MGC192.168.206.145@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6522.010257] Lustre: 165333:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.145@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6527.457613] LustreError: 165338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6527.481827] LustreError: 165338:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6559.461800] LustreError: 165355:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6559.475741] LustreError: 165355:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 25 previous similar messages [ 6582.983424] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6582.990139] Lustre: 165333:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.145@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6623.740141] LustreError: 165338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6623.754050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6623.771056] LustreError: 165338:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 41 previous similar messages [ 6628.844196] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6629.038336] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6638.395218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6653.423076] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 6662.912461] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6663.177981] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6671.421320] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6673.486191] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6680.535310] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6682.784751] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6704.109206] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6704.118024] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6704.120915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6704.137789] Lustre: Skipped 3 previous similar messages [ 6704.163682] Lustre: Skipped 5 previous similar messages [ 6705.543406] Lustre: server umount lustre-MDT0000 complete [ 6709.507757] LustreError: 161475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836854 with bad export cookie 12141929017021741399 [ 6709.513558] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6709.522361] LustreError: 161475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6709.886859] Lustre: server umount lustre-MDT0001 complete [ 6724.635663] Lustre: server umount lustre-OST0000 complete [ 6749.184384] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 6759.747619] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 6801.508771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6801.777907] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6801.809602] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6801.875807] Lustre: lustre-MDT0000: new disk, initializing [ 6801.993348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6802.001162] Lustre: Skipped 1 previous similar message [ 6802.035762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6806.873644] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6818.096411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6818.297412] Lustre: 170443:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6818.581592] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6823.316848] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6832.233201] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6838.803691] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6846.137871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6846.424608] Lustre: lustre-OST0000: new disk, initializing [ 6846.427685] Lustre: Skipped 1 previous similar message [ 6846.430938] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6846.436286] Lustre: Skipped 2 previous similar messages [ 6846.441354] Lustre: 172412:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6852.709819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6852.715357] Lustre: Skipped 1 previous similar message [ 6852.722090] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6852.754635] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6852.794712] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6861.758646] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6866.325216] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6866.615643] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6872.068444] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6872.334737] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6878.689130] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6878.691129] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6878.699746] LustreError: Skipped 1 previous similar message [ 6878.703578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6878.722665] Lustre: Skipped 1 previous similar message [ 6878.746073] Lustre: Skipped 1 previous similar message [ 6880.815562] Lustre: server umount lustre-OST0000 complete [ 6891.393657] Lustre: server umount lustre-MDT0000 complete [ 6894.055649] LustreError: 172420:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6894.094487] LustreError: 172420:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 32 previous similar messages [ 6895.537807] LustreError: 170436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837040 with bad export cookie 12141929017021758206 [ 6895.550325] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6895.553709] LustreError: 170436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6895.874860] Lustre: server umount lustre-MDT0001 complete [ 6908.131658] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 09:24:09 (1787837049) [ 6916.271030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6923.037198] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6932.447211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6936.840951] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6945.231771] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6951.254493] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6957.892323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6964.479085] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6972.383549] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6985.608180] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 09:25:28 (1787837128) [ 6992.353824] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6992.359255] LustreError: Skipped 1 previous similar message [ 6992.362021] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6992.374172] Lustre: Skipped 2 previous similar messages [ 6992.378894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6992.384123] Lustre: Skipped 3 previous similar messages [ 6995.220626] Lustre: server umount lustre-OST0000 complete [ 7006.910862] Lustre: server umount lustre-MDT0000 complete [ 7011.491268] LustreError: 174804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837156 with bad export cookie 12141929017021759088 [ 7011.504768] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7011.512846] LustreError: 174804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7012.150896] Lustre: server umount lustre-MDT0001 complete [ 7024.072486] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 7027.049408] Key type lgssc unregistered [ 7027.250759] LNet: 178558:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7027.262168] LNetError: 178558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7027.283354] LNet: Removed LNI 192.168.206.145@tcp [ 7028.083844] Key type .llcrypt unregistered [ 7028.089058] Key type ._llcrypt unregistered [ 7046.627506] Key type ._llcrypt registered [ 7046.629537] Key type .llcrypt registered [ 7046.726275] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 7047.628888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7047.665903] alg: No test for adler32 (adler32-zlib) [ 7048.739102] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 7049.018976] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 7050.704655] Key type lgssc registered [ 7051.602455] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7058.933503] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7058.974858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7074.591439] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7079.713123] LustreError: 179803:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.145@tcp: failed processing log, type 4: rc = -110 [ 7084.836750] LustreError: 179803:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7084.845876] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7084.850533] LustreError: MGC192.168.206.145@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7084.857193] LustreError: 179803:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7084.864544] LustreError: 179803:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7084.869513] LustreError: 179803:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7084.928673] Lustre: server umount lustre-OST0000 complete [ 7084.931677] LustreError: 179803:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7093.199576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7094.847692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7099.797559] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7104.996695] LustreError: 180305:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7105.016733] LustreError: 180305:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7108.715055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7108.985746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7112.960702] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7120.793248] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7125.979211] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7131.604168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7131.834323] LustreError: 182119:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7131.842440] Lustre: 182119:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.206.145@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7131.991282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7137.376826] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7145.071493] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7149.544872] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7149.549932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7149.558318] Lustre: Skipped 1 previous similar message [ 7149.579949] Lustre: Skipped 2 previous similar messages [ 7154.661135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7154.673195] Lustre: Skipped 2 previous similar messages [ 7159.780379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7159.795158] Lustre: Skipped 2 previous similar messages [ 7161.311200] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7161.471764] Lustre: server umount lustre-MDT0000 complete [ 7164.898664] LustreError: 181352:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7164.918204] LustreError: 181352:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7165.204082] LustreError: 182146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837310 with bad export cookie 10385614362576276036 [ 7165.223827] LustreError: 182146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7165.229085] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7165.636977] Lustre: server umount lustre-MDT0001 complete [ 7180.430382] Lustre: server umount lustre-OST0000 complete [ 7190.332572] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 7200.091304] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 7234.101496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7234.289423] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7234.310150] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7234.387721] Lustre: lustre-MDT0000: new disk, initializing [ 7234.467249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7234.480305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7238.251678] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7248.635990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7248.731169] Lustre: 185646:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7248.773525] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7248.777138] Lustre: Skipped 1 previous similar message [ 7248.832217] Lustre: lustre-MDT0001: new disk, initializing [ 7248.906505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7248.934289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7248.947963] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7253.075236] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7261.996685] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7267.541521] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7273.827332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7274.069917] Lustre: lustre-OST0000: new disk, initializing [ 7274.074876] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7274.080602] Lustre: 187613:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7274.148336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7279.937822] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7280.192819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7280.201827] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7280.331188] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7288.470543] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7292.836248] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7293.092972] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7297.166645] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7297.325423] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7301.088948] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7301.090937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7301.102286] Lustre: Skipped 2 previous similar messages [ 7301.120525] Lustre: Skipped 1 previous similar message [ 7306.210502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7306.220961] Lustre: Skipped 1 previous similar message [ 7313.375250] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7313.558575] Lustre: server umount lustre-OST0000 complete [ 7320.545045] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7320.556160] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7320.571629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7320.579265] Lustre: Skipped 2 previous similar messages [ 7323.673086] Lustre: server umount lustre-MDT0000 complete [ 7326.688590] LustreError: 185653:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7326.723215] LustreError: 185653:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7327.016486] LustreError: 186439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837471 with bad export cookie 10385614362576276757 [ 7327.020546] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7327.039133] LustreError: 186439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7327.338996] Lustre: server umount lustre-MDT0001 complete [ 7337.630876] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 09:31:19 (1787837479) [ 7355.212413] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 7357.581176] Key type lgssc unregistered [ 7357.832811] LNet: 190561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7357.844674] LNetError: 190561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7357.867304] LNet: Removed LNI 192.168.206.145@tcp [ 7358.816360] Key type .llcrypt unregistered [ 7358.820612] Key type ._llcrypt unregistered [ 7375.286151] Key type ._llcrypt registered [ 7375.287700] Key type .llcrypt registered [ 7375.396500] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 7376.342836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7376.372808] alg: No test for adler32 (adler32-zlib) [ 7377.388649] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 7377.665665] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 7379.359195] Key type lgssc registered [ 7380.365850] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7387.176681] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7387.194438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7388.726094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7393.315754] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7398.883184] LustreError: 191777:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7398.912856] LustreError: 191777:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7401.694334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7401.954933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7405.394941] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7411.816752] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7416.866662] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7423.536172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7423.741903] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7423.744964] LustreError: 193642:0:(llog.c:1696:llog_backup()) MGC192.168.206.145@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7423.754242] Lustre: 193642:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.206.145@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7423.859607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7429.549152] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7436.356095] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7439.328916] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7439.334863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7439.347697] Lustre: Skipped 2 previous similar messages [ 7439.352170] Lustre: Skipped 1 previous similar message [ 7444.448728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7444.457732] Lustre: Skipped 2 previous similar messages [ 7444.730925] Lustre: server umount lustre-MDT0000 complete [ 7447.815941] LustreError: 191758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837592 with bad export cookie 13382561220831088443 [ 7447.826451] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7447.829680] LustreError: 191758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7448.152588] Lustre: server umount lustre-MDT0001 complete [ 7462.180710] Lustre: server umount lustre-OST0000 complete [ 7471.504792] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 7480.886741] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 7516.964927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7517.197195] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7517.220577] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7517.292137] Lustre: lustre-MDT0000: new disk, initializing [ 7517.341449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7517.366144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7521.871405] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7531.458659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7531.542047] Lustre: 197166:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7531.575125] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7531.577150] Lustre: Skipped 1 previous similar message [ 7531.579722] LustreError: 197948:0:(llog.c:1696:llog_backup()) MGC192.168.206.145@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7531.584869] LustreError: 197948:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7531.588964] Lustre: 197948:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.145@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7531.601791] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7531.606786] Lustre: Skipped 1 previous similar message [ 7531.688649] Lustre: lustre-MDT0001: new disk, initializing [ 7531.807234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7531.839858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7531.861493] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7535.829710] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7545.199794] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7550.990899] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7557.747422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7557.991912] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7557.994461] Lustre: Skipped 1 previous similar message [ 7557.997789] LustreError: 199111:0:(llog.c:1696:llog_backup()) MGC192.168.206.145@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7558.003460] LustreError: 199111:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7558.008195] Lustre: 199111:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.145@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7558.016320] Lustre: 199111:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7558.034238] Lustre: lustre-OST0000: new disk, initializing [ 7558.038173] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7558.042885] Lustre: 199135:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7558.120118] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7563.809244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7563.819992] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7563.891127] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7564.104301] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7573.239361] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7578.222128] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7578.430900] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7583.148777] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7583.394678] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7589.350180] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7589.359669] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7589.365127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7589.365878] LustreError: Skipped 1 previous similar message [ 7589.402239] Lustre: Skipped 1 previous similar message [ 7594.474940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7594.481324] Lustre: Skipped 2 previous similar messages [ 7599.598092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7599.607171] Lustre: Skipped 1 previous similar message [ 7600.099503] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7600.276469] Lustre: server umount lustre-OST0000 complete [ 7604.708676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7604.717920] Lustre: Skipped 1 previous similar message [ 7609.828477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7609.840891] Lustre: Skipped 3 previous similar messages [ 7610.767902] Lustre: server umount lustre-MDT0000 complete [ 7614.953329] LustreError: 197178:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7614.979137] LustreError: 197178:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7615.009024] LustreError: 197960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837759 with bad export cookie 13382561220831089157 [ 7615.020530] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7615.025060] LustreError: 197960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7615.475200] Lustre: server umount lustre-MDT0001 complete [ 7624.953811] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 09:36:07 (1787837767) [ 7640.894682] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 7650.053903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7650.400312] LustreError: 202264:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7650.444967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7655.147398] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7655.904167] LustreError: 202265:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7661.029550] LustreError: 202264:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7663.507596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7668.203683] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7671.273807] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7678.287104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7684.472261] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7692.060140] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7692.185612] Lustre: lustre-OST0001: new disk, initializing [ 7692.188770] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7692.193612] Lustre: 204572:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7692.251627] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7692.258864] Lustre: Skipped 2 previous similar messages [ 7698.002802] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7698.006673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7698.028498] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7698.154989] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7708.665537] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7720.161365] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7722.045779] Lustre: DEBUG MARKER: ADD OST3 [ 7727.020812] loop: module loaded [ 7727.092539] loop0: detected capacity change from 0 to 204800000 [ 7727.264817] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7727.271537] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7727.280566] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7733.524468] loop0: detected capacity change from 0 to 204800000 [ 7734.495846] Lustre: DEBUG MARKER: STOP OST3 [ 7739.132675] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7739.329278] Lustre: lustre-OST0003: new disk, initializing [ 7739.332657] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7739.342465] Lustre: 206580:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7741.296077] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7741.432305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7741.444459] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7741.480770] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7742.661332] Lustre: Failing over lustre-OST0003 [ 7742.776932] Lustre: server umount lustre-OST0003 complete [ 7743.329764] LustreError: 206774:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7745.246081] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7746.314573] LustreError: 203751:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.206.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7746.529266] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7746.538230] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7746.551752] Lustre: Skipped 1 previous similar message [ 7753.202964] Lustre: DEBUG MARKER: START OST3 again [ 7759.464370] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7759.826320] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7759.831296] Lustre: Skipped 1 previous similar message [ 7759.858848] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7760.957870] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7761.816303] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7761.817131] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7765.518850] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7785.367917] Lustre: Failing over lustre-OST0003 [ 7785.447922] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7785.457176] LustreError: 203751:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7785.465094] LustreError: 203751:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 7785.487841] Lustre: server umount lustre-OST0003 complete [ 7805.927021] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7805.931073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7805.945709] Lustre: Skipped 3 previous similar messages [ 7805.954041] Lustre: Skipped 1 previous similar message [ 7809.197042] Lustre: server umount lustre-MDT0000 complete [ 7812.862912] LustreError: 206575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837957 with bad export cookie 13382561220831090046 [ 7812.885957] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7812.890034] LustreError: 206575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7819.362399] Lustre: server umount lustre-MDT0001 complete [ 7823.195240] Lustre: server umount lustre-OST0000 complete [ 7827.655965] Lustre: server umount lustre-OST0001 complete [ 7835.336584] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 7843.230766] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 7881.905168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7882.162257] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7882.187426] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7882.275773] Lustre: lustre-MDT0000: new disk, initializing [ 7882.353516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7887.281204] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7897.475982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7897.569331] Lustre: 212032:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7897.795333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7897.801185] Lustre: Skipped 1 previous similar message [ 7897.828523] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7902.137746] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7911.429992] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7917.887527] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7926.921965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7927.145427] Lustre: lustre-OST0000: new disk, initializing [ 7927.148644] Lustre: Skipped 1 previous similar message [ 7927.154699] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7927.162775] Lustre: Skipped 2 previous similar messages [ 7927.168528] Lustre: 214003:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7929.107065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7929.116606] Lustre: Skipped 1 previous similar message [ 7929.127337] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7929.293543] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7933.567500] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7942.791488] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7946.622895] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7946.830584] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7950.646896] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7950.854413] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7954.912409] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7954.918408] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7954.922946] LustreError: Skipped 1 previous similar message [ 7954.924240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7954.933029] Lustre: Skipped 4 previous similar messages [ 7954.965812] Lustre: Skipped 7 previous similar messages [ 7958.563182] Lustre: server umount lustre-OST0000 complete [ 7964.642034] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7968.025800] Lustre: server umount lustre-MDT0000 complete [ 7970.279443] LustreError: 212041:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7970.301524] LustreError: 212041:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [ 7971.570168] LustreError: 212023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838116 with bad export cookie 13382561220831092335 [ 7971.585269] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7971.589521] LustreError: 212023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7971.934254] Lustre: server umount lustre-MDT0001 complete [ 7975.625796] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 09:41:58 (1787838118) [ 7991.294813] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 8000.865242] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 8036.200453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8036.414648] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8036.437082] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8036.530674] Lustre: lustre-MDT0000: new disk, initializing [ 8036.740678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8042.272587] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8054.706535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8054.857715] Lustre: 218614:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8055.179704] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8059.711152] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8069.878564] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8075.565359] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8082.309440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8082.686519] Lustre: 220586:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8084.765096] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8088.520375] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8099.219707] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8105.129441] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8105.428551] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8110.134133] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8110.395334] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8115.683371] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8115.685702] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8115.698854] LustreError: Skipped 1 previous similar message [ 8115.707895] Lustre: Skipped 2 previous similar messages [ 8115.712307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8115.718162] Lustre: Skipped 4 previous similar messages [ 8118.378135] Lustre: server umount lustre-OST0000 complete [ 8128.227082] Lustre: server umount lustre-MDT0000 complete [ 8131.045385] LustreError: 218626:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8131.077071] LustreError: 218626:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8132.745811] LustreError: 218608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838277 with bad export cookie 13382561220831093224 [ 8132.746671] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8132.762954] LustreError: 218608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8133.201648] Lustre: server umount lustre-MDT0001 complete [ 8142.311589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8147.757659] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8156.299160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8156.718171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8156.724054] Lustre: Skipped 5 previous similar messages [ 8161.483634] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8170.469403] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8177.179102] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8185.345964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8192.163638] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8200.567900] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8211.428340] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8211.435647] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8211.452936] Lustre: Skipped 2 previous similar messages [ 8216.208438] Lustre: server umount lustre-OST0000 complete [ 8226.155228] Lustre: server umount lustre-MDT0000 complete [ 8229.798318] LustreError: 222782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838374 with bad export cookie 13382561220831094099 [ 8229.806360] LustreError: 222782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8229.808896] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8230.073063] Lustre: server umount lustre-MDT0001 complete [ 8338.411238] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 8340.786488] Key type lgssc unregistered [ 8341.143420] LNet: 226469:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8341.153954] LNetError: 226469:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8341.170702] LNet: Removed LNI 192.168.206.145@tcp [ 8341.917152] Key type .llcrypt unregistered [ 8341.919207] Key type ._llcrypt unregistered [ 8364.259407] Key type ._llcrypt registered [ 8364.261863] Key type .llcrypt registered [ 8364.344273] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 8378.974748] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 8380.449423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8380.566162] alg: No test for adler32 (adler32-zlib) [ 8381.651299] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 8381.928925] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 8383.671156] Key type lgssc registered [ 8384.833784] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8421.323184] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8421.360244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8422.565932] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8422.625549] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8422.706476] Lustre: lustre-MDT0000: new disk, initializing [ 8422.778097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8422.800118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8426.741517] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8436.909132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8437.085936] Lustre: 229823:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8437.144183] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8437.148436] Lustre: Skipped 1 previous similar message [ 8437.230499] Lustre: lustre-MDT0001: new disk, initializing [ 8437.335754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8437.366233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8437.374834] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8441.150879] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8450.283205] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8456.177161] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8464.184276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8464.532466] Lustre: lustre-OST0000: new disk, initializing [ 8464.535966] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8464.543304] Lustre: 231790:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8464.652818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8471.795429] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8474.211748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8474.222237] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8474.367833] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8482.247236] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8487.387415] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8487.597130] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8492.138356] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8492.385862] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8495.088169] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8495.089091] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8495.101362] LustreError: Skipped 1 previous similar message [ 8495.102457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8495.130594] Lustre: Skipped 1 previous similar message [ 8500.197162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8500.201330] Lustre: Skipped 2 previous similar messages [ 8500.563827] Lustre: server umount lustre-OST0000 complete [ 8505.315762] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8505.319853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8505.335988] Lustre: Skipped 1 previous similar message [ 8505.354436] Lustre: Skipped 1 previous similar message [ 8510.445555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8510.455278] Lustre: Skipped 1 previous similar message [ 8510.597793] Lustre: server umount lustre-MDT0000 complete [ 8514.334465] LustreError: 229815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838659 with bad export cookie 2200952926284785883 [ 8514.341511] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8514.347859] LustreError: 229815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8514.690616] Lustre: server umount lustre-MDT0001 complete [ 8534.303165] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 8544.634285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8545.041487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8549.397338] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8550.370565] LustreError: 234724:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8550.393862] LustreError: 234724:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8554.471525] LustreError: 234725:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8559.598401] LustreError: 234724:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8559.640349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8560.087358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8565.492969] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8568.816707] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8576.493548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8576.923716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8583.738652] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8593.253625] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8593.486511] Lustre: lustre-OST0001: new disk, initializing [ 8593.491236] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8593.496841] Lustre: 237034:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8593.651790] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8600.885451] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8601.752831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8601.771874] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8601.830573] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8612.522698] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8623.863482] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8637.714642] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 09:53:00 (1787838780) [ 8642.021637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8642.033068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8642.049682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8646.158248] Lustre: server umount lustre-MDT0000 complete [ 8648.176911] LustreError: 234725:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8648.205544] LustreError: 234725:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8650.024671] LustreError: 234706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838794 with bad export cookie 2200952926284786765 [ 8650.026970] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8650.044496] LustreError: 234706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8650.698241] Lustre: server umount lustre-MDT0001 complete [ 8664.950189] Lustre: server umount lustre-OST0000 complete [ 8679.088424] Lustre: server umount lustre-OST0001 complete [ 8685.893678] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 8696.495669] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 8733.626177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8733.893785] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8733.923295] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8734.046717] Lustre: lustre-MDT0000: new disk, initializing [ 8734.141391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8734.156910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8739.283889] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8750.104362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8750.255390] Lustre: 240979:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8750.320541] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8750.323693] Lustre: Skipped 1 previous similar message [ 8750.408593] Lustre: lustre-MDT0001: new disk, initializing [ 8750.664661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8750.679663] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8755.958570] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8765.164133] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8770.355719] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8777.276057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8777.630567] Lustre: lustre-OST0000: new disk, initializing [ 8777.635617] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8777.642514] Lustre: 242948:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8778.766386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8778.780604] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8778.991915] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8783.994691] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8793.467151] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8798.820521] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8799.061883] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8803.495732] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8803.744720] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8809.952359] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8809.954430] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8809.964164] LustreError: Skipped 1 previous similar message [ 8809.972328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8809.972340] Lustre: Skipped 4 previous similar messages [ 8810.015832] Lustre: Skipped 4 previous similar messages [ 8820.193886] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8820.363780] Lustre: server umount lustre-OST0000 complete [ 8825.316315] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8825.328742] Lustre: Skipped 1 previous similar message [ 8830.437365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8830.445826] Lustre: Skipped 8 previous similar messages [ 8830.638274] Lustre: server umount lustre-MDT0000 complete [ 8835.393792] LustreError: 240972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838980 with bad export cookie 2200952926284788900 [ 8835.396133] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8835.403264] LustreError: 240972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8835.821915] Lustre: server umount lustre-MDT0001 complete [ 8845.332429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8846.032654] LustreError: 245165:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8846.135250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8846.141412] Lustre: Skipped 2 previous similar messages [ 8852.092632] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8855.554392] LustreError: 245166:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8855.573323] LustreError: 245166:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8861.046750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8866.100368] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8873.518276] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8879.482473] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8886.702284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8893.432698] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8901.379948] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9050.082757] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9050.086167] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9050.105648] LustreError: Skipped 1 previous similar message [ 9050.107765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9050.124017] Lustre: Skipped 1 previous similar message [ 9050.147801] Lustre: Skipped 1 previous similar message [ 9052.375516] Lustre: server umount lustre-OST0000 complete [ 9056.227543] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9062.193919] Lustre: server umount lustre-MDT0000 complete [ 9065.168143] LustreError: 245147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839210 with bad export cookie 2200952926284789782 [ 9065.173296] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9065.176245] LustreError: 245147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9065.460639] Lustre: server umount lustre-MDT0001 complete [ 9169.867507] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 9172.838188] Key type lgssc unregistered [ 9173.262580] LNet: 248787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9173.285475] LNetError: 248787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9173.313168] LNet: Removed LNI 192.168.206.145@tcp [ 9174.256342] Key type .llcrypt unregistered [ 9174.258669] Key type ._llcrypt unregistered [ 9201.817351] Key type ._llcrypt registered [ 9201.819058] Key type .llcrypt registered [ 9202.001649] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 9215.490959] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 9216.277758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9216.331691] alg: No test for adler32 (adler32-zlib) [ 9217.414685] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 9217.624613] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 9219.266471] Key type lgssc registered [ 9220.888416] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9267.618859] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 9277.463708] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9277.511381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9278.721238] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9278.766871] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9278.837230] Lustre: lustre-MDT0000: new disk, initializing [ 9278.887045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9278.899855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9282.649768] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 9291.839869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9291.950373] Lustre: 252837:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9292.014694] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9292.018592] Lustre: Skipped 1 previous similar message [ 9292.073705] Lustre: lustre-MDT0001: new disk, initializing [ 9292.142463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9292.185173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9292.207790] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9296.486134] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 9300.771928] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9306.603312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9306.853666] Lustre: lustre-OST0000: new disk, initializing [ 9306.856981] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9306.867428] Lustre: 254467:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9306.948555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9307.512437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9307.518767] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9307.648134] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9312.846525] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 9321.904794] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9322.232268] Lustre: lustre-OST0001: new disk, initializing [ 9322.239512] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9322.254075] Lustre: 255341:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9322.403160] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9328.784954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9328.801052] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9329.069598] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9330.545691] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 9342.899673] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9349.110795] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9359.845623] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9359.850619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9359.859572] Lustre: Skipped 3 previous similar messages [ 9359.871343] Lustre: Skipped 3 previous similar messages [ 9364.964552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9364.970484] Lustre: Skipped 1 previous similar message [ 9365.804745] Lustre: server umount lustre-MDT0000 complete [ 9369.120794] LustreError: 252830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839513 with bad export cookie 7331297056259304637 [ 9369.122380] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9369.130648] LustreError: 252830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9369.462843] Lustre: server umount lustre-MDT0001 complete [ 9383.625387] Lustre: server umount lustre-OST0000 complete [ 9385.423159] Lustre: 250346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787839514/real 1787839514] req@ffff99c58362f0c0 x1874685446923776/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787839530 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9385.451969] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9386.828269] Lustre: server umount lustre-OST0001 complete [ 9401.199755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9411.401306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9419.086944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9428.500716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9450.114069] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 9461.812145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9461.830686] Lustre: lustre-MDT0000: reset Object Index mappings [ 9462.161427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9466.939137] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 9467.362300] LustreError: 260352:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9467.377469] LustreError: 260352:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9471.463841] LustreError: 260353:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9475.035302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9475.057300] Lustre: lustre-MDT0001: reset Object Index mappings [ 9475.507649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9479.043716] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 9481.967352] Lustre: 261493:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9488.635651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9495.025722] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 9500.270914] LustreError: 261847:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9502.970518] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9503.207594] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9503.212059] Lustre: Skipped 1 previous similar message [ 9506.311240] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9506.417120] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9508.940883] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 9516.244147] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9519.846578] Lustre: 263365:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9526.393511] Lustre: server umount lustre-MDT0000 complete [ 9526.753346] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9526.758614] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9526.761968] LustreError: 262157:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9526.780459] Lustre: Skipped 3 previous similar messages [ 9526.817804] LustreError: 262157:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9529.751443] LustreError: 260332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839674 with bad export cookie 7331297056259311427 [ 9529.762517] LustreError: 260332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9529.762881] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9530.096656] Lustre: server umount lustre-MDT0001 complete [ 9544.253031] Lustre: server umount lustre-OST0000 complete [ 9547.235609] Lustre: 250344:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787839676/real 1787839676] req@ffff99c5a1e50000 x1874685446970752/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787839692 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9547.262658] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9548.462481] Lustre: server umount lustre-OST0001 complete [ 9567.524804] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 10:08:30 (1787839710) [ 9582.470852] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 9592.119967] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 9621.121963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9621.327201] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9621.366662] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9621.435639] Lustre: lustre-MDT0000: new disk, initializing [ 9621.503198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9621.521518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9625.204664] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 9635.325364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9635.428363] Lustre: 267617:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9635.477862] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9635.482079] Lustre: Skipped 1 previous similar message [ 9635.558104] Lustre: lustre-MDT0001: new disk, initializing [ 9635.670176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9635.681754] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9639.737818] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 9648.697260] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9654.264242] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9660.595601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9660.831833] Lustre: lustre-OST0000: new disk, initializing [ 9660.836552] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9660.844655] Lustre: 269586:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9662.711199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9662.724776] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9662.834569] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9666.454834] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 9675.311567] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9680.065320] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9680.259478] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9684.509205] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9684.745239] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9688.544887] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9688.551554] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9688.595222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9688.612387] Lustre: Skipped 1 previous similar message [ 9693.679024] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9693.683209] Lustre: Skipped 1 previous similar message [ 9698.787083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9698.801321] Lustre: Skipped 1 previous similar message [ 9701.345781] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9701.553438] Lustre: server umount lustre-OST0000 complete [ 9707.488272] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9707.503189] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9707.521367] Lustre: Skipped 1 previous similar message [ 9707.526149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9712.544357] Lustre: server umount lustre-MDT0000 complete [ 9714.144436] LustreError: 267627:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9714.162860] LustreError: 267627:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9716.897519] LustreError: 268423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839861 with bad export cookie 7331297056259313513 [ 9716.905271] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9716.910404] LustreError: 268423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9717.227443] Lustre: server umount lustre-MDT0001 complete [ 9736.068754] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 9746.005305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9746.432708] LustreError: 272522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9746.491749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9746.501109] Lustre: Skipped 2 previous similar messages [ 9751.105331] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 9760.262730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9765.363561] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 9768.742888] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9774.320245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9779.848744] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 9786.615941] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9786.772242] Lustre: lustre-OST0001: new disk, initializing [ 9786.775552] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9786.782036] Lustre: 274826:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9788.700494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9788.724473] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9788.830306] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9791.964968] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 9800.009214] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9803.440288] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9808.789623] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9808.894769] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9817.062194] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9817.070616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9817.090385] Lustre: Skipped 1 previous similar message [ 9817.103123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9817.115902] Lustre: Skipped 2 previous similar messages [ 9820.653365] Lustre: server umount lustre-MDT0000 complete [ 9822.692471] LustreError: 275357:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9822.717344] LustreError: 275357:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9824.474682] LustreError: 274825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839969 with bad export cookie 7331297056259314402 [ 9824.503689] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9824.759960] Lustre: server umount lustre-MDT0001 complete [ 9838.762987] Lustre: server umount lustre-OST0000 complete [ 9851.678677] Lustre: server umount lustre-OST0001 complete [ 9863.548649] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 9871.179356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9871.407458] LustreError: 277884:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9875.255645] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 9881.923118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9882.081646] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9882.090312] Lustre: Skipped 3 previous similar messages [ 9882.274876] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9882.279167] Lustre: Skipped 4 previous similar messages [ 9885.799626] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 9888.758734] Lustre: 279025:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9894.684979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9897.092696] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9900.828558] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 9909.408810] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9914.885246] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9916.358680] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 9924.379592] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9927.746658] Lustre: 280902:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9932.910710] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9932.966115] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9941.540513] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 10:14:44 (1787840084) [ 9955.814815] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9955.815827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9955.825612] Lustre: Skipped 5 previous similar messages [ 9955.835870] Lustre: Skipped 1 previous similar message [ 9957.636837] Lustre: server umount lustre-MDT0000 complete [ 9960.663974] LustreError: 280904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840105 with bad export cookie 7331297056259320751 [ 9960.667535] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9960.676666] LustreError: 280904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9960.931362] LustreError: 277879:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9960.943374] LustreError: 277879:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9967.029151] Lustre: server umount lustre-MDT0001 complete [ 9976.353955] Lustre: server umount lustre-OST0000 complete [ 9985.819439] Lustre: server umount lustre-OST0001 complete [ 9990.858765] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [10000.197054] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [10042.612283] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [10053.514298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10053.847491] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10053.875028] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10054.012843] Lustre: lustre-MDT0000: new disk, initializing [10054.132616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10059.094647] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [10072.092401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10072.230370] Lustre: 285143:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10072.568367] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10077.239843] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [10088.266079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10088.518426] Lustre: 286776:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10090.202805] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10094.267369] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [10103.854871] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10104.086479] Lustre: 287648:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10105.371852] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10105.386278] Lustre: Skipped 1 previous similar message [10105.448290] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10109.816436] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [10118.766829] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10121.965646] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10121.974019] Lustre: Skipped 1 previous similar message [10135.402376] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 10:17:57 (1787840277) [10139.105412] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10139.114498] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10139.121608] Lustre: Skipped 3 previous similar messages [10139.127991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10139.134623] Lustre: Skipped 6 previous similar messages [10139.388560] Lustre: server umount lustre-MDT0000 complete [10141.161050] LustreError: 285153:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10141.188465] LustreError: 285153:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10142.301068] LustreError: 285133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840287 with bad export cookie 7331297056259328584 [10142.312681] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10142.313054] LustreError: 285133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10142.640517] Lustre: server umount lustre-MDT0001 complete [10155.544297] Lustre: server umount lustre-OST0000 complete [10170.113493] Lustre: server umount lustre-OST0001 complete [10179.572312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10180.455661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10180.460175] Lustre: Skipped 6 previous similar messages [10185.028336] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [10193.646313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10198.345218] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [10205.301851] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10210.594494] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10217.232116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10223.729230] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [10231.106588] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10247.768444] loop1: detected capacity change from 0 to 204800000 [10247.994124] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10248.001886] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10248.010340] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10253.230430] loop1: detected capacity change from 0 to 204800000 [10259.562320] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10259.654575] LustreError: 289878:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs9293: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10259.673908] LustreError: 289878:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs9293 [10259.679156] LustreError: 289878:0:(mgs_handler.c:517:mgs_target_reg()) fs9293-MDT0001: Error checking target index: rc = -13 [10259.702478] LustreError: fs9293-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10259.713407] LustreError: 293612:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10259.719866] LustreError: 293612:0:(tgt_mount.c:2129:server_put_super()) no obd fs9293-MDT0001 [10259.724054] LustreError: 293612:0:(tgt_mount.c:116:server_deregister_mount()) fs9293-MDT0001 not registered [10259.750509] Lustre: server umount fs9293-MDT0001 complete [10259.753485] LustreError: 293612:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10268.818650] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10268.920243] Lustre: Setting parameter fs9293-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs9293-MDT0001 [10268.958707] Lustre: srv-fs9293-MDT0001: No data found on store. Initialize space. [10268.962138] Lustre: Skipped 4 previous similar messages [10269.049689] Lustre: fs9293-MDT0001: new disk, initializing [10269.053297] Lustre: Skipped 3 previous similar messages [10269.163195] Lustre: srv-fs9293-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10269.169873] Lustre: 294079:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs9293-MDT0001-mdtlov: can't create new llog: rc = -115 [10269.177095] LustreError: 294079:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs9293-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10273.346455] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [10279.628618] Lustre: server umount fs9293-MDT0001 complete [10290.891143] loop1: detected capacity change from 0 to 204800000 [10295.085129] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10295.347527] Lustre: 295645:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs9293-MDT0001-mdtlov: can't create new llog: rc = -115 [10295.362421] LustreError: 295645:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs9293-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10299.047930] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [10302.956893] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10303.281932] Lustre: server umount fs9293-MDT0001 complete [10313.187168] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10313.198866] Lustre: Skipped 5 previous similar messages [10318.151633] Lustre: server umount lustre-MDT0000 complete [10325.555958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10325.689635] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10329.876114] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [10331.130085] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10336.648626] loop1: detected capacity change from 0 to 204800000 [10342.565233] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10342.852166] Lustre: 298089:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs9293-MDT0001-mdtlov: can't create new llog: rc = -115 [10342.863546] LustreError: 298089:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs9293-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10347.341288] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [10352.034952] Lustre: server umount fs9293-MDT0001 complete [10377.200514] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10377.217509] LustreError: Skipped 1 previous similar message [10382.657706] Lustre: server umount lustre-OST0000 complete [10392.330089] Lustre: server umount lustre-MDT0000 complete [10396.111518] LustreError: 289871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840540 with bad export cookie 7331297056259331265 [10396.130067] LustreError: 289871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10396.130635] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10396.520333] Lustre: server umount lustre-MDT0001 complete [10409.789274] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [10414.045620] Key type lgssc unregistered [10414.347887] LNet: 300439:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10414.355032] LNetError: 300439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10414.380702] LNet: Removed LNI 192.168.206.145@tcp [10415.741350] Key type .llcrypt unregistered [10415.747382] Key type ._llcrypt unregistered [10436.600481] Key type ._llcrypt registered [10436.605362] Key type .llcrypt registered [10436.698241] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [10437.435719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10437.516443] alg: No test for adler32 (adler32-zlib) [10438.601259] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [10438.848702] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [10440.519200] Key type lgssc registered [10441.590249] Lustre: Echo OBD driver; http://www.lustre.org/ [10449.193777] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10449.220686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10450.606082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10454.629476] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [10461.155126] LustreError: 301713:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10461.165161] LustreError: 301713:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10462.359800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10462.851553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10466.558605] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [10473.012721] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10477.870399] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10483.869073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10484.278252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10487.354212] LustreError: 303543:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10489.335298] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [10492.396836] LustreError: 303541:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10495.186468] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10496.484947] LustreError: 303542:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10496.508721] LustreError: 303542:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10500.873798] LustreError: 303543:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10511.075315] LustreError: 303543:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10511.094957] LustreError: 303543:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10514.028288] loop1: detected capacity change from 0 to 204800000 [10514.227985] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10514.245561] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10514.261082] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10519.701212] loop1: detected capacity change from 0 to 204800000 [10524.830857] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10524.943470] Lustre: fs4693-OST0000: new disk, initializing [10524.947381] Lustre: srv-fs4693-OST0000: No data found on store. Initialize space. [10524.955689] Lustre: 305446:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10525.025814] Lustre: fs4693-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10527.206862] LustreError: 303543:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10527.224843] LustreError: 303543:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [10529.705957] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [10536.775873] Lustre: server umount fs4693-OST0000 complete [10546.070684] loop1: detected capacity change from 0 to 204800000 [10550.196582] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10550.393236] Lustre: fs4693-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10554.932931] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [10561.305815] loop2: detected capacity change from 0 to 204800000 [10561.510899] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10561.525362] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10561.536395] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10562.276098] LustreError: 303541:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10562.311801] LustreError: 303541:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [10565.742640] loop2: detected capacity change from 0 to 204800000 [10570.012882] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10570.070869] LustreError: fs4693-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10570.080122] LustreError: fs4693-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10570.088651] LustreError: 308485:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10570.093297] LustreError: 308485:0:(tgt_mount.c:2129:server_put_super()) no obd fs4693-OST0001 [10570.097631] LustreError: 308485:0:(tgt_mount.c:116:server_deregister_mount()) fs4693-OST0001 not registered [10570.119675] Lustre: server umount fs4693-OST0001 complete [10570.121478] LustreError: 308485:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10573.293633] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10578.096213] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10578.348172] Lustre: fs4693-OST0001: new disk, initializing [10578.352059] Lustre: srv-fs4693-OST0001: No data found on store. Initialize space. [10578.358417] Lustre: 309037:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10578.421743] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10578.482419] Lustre: fs4693-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10584.530725] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [10595.244519] Lustre: server umount fs4693-OST0000 complete [10606.679172] Lustre: server umount fs4693-OST0001 complete [10619.865133] loop1: detected capacity change from 0 to 204800000 [10625.575805] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10625.721508] Lustre: fs4693-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10627.555374] LustreError: 303542:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10627.576345] LustreError: 303542:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 39 previous similar messages [10631.253094] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [10637.287371] loop2: detected capacity change from 0 to 204800000 [10643.456041] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10649.548918] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [10658.464166] Lustre: server umount fs4693-OST0000 complete [10665.443042] Lustre: server umount fs4693-OST0001 complete [10676.714756] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10676.724923] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10676.738654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10678.761283] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10678.766987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10678.779035] Lustre: Skipped 1 previous similar message [10682.218276] Lustre: server umount lustre-OST0000 complete [10687.975527] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10687.987404] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10688.011670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10691.253446] Lustre: server umount lustre-MDT0000 complete [10693.915327] LustreError: 301695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840838 with bad export cookie 6010554659272374591 [10693.916757] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10693.925156] LustreError: 301695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10694.192151] Lustre: server umount lustre-MDT0001 complete [10703.928706] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [10707.074768] Key type lgssc unregistered [10707.294082] LNet: 314961:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10707.310372] LNetError: 314961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10707.323249] LNet: Removed LNI 192.168.206.145@tcp [10707.999186] Key type .llcrypt unregistered [10708.000824] Key type ._llcrypt unregistered [10735.422077] Key type ._llcrypt registered [10735.424516] Key type .llcrypt registered [10735.511554] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [10747.525726] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [10748.178186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10748.236597] alg: No test for adler32 (adler32-zlib) [10749.219294] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [10749.411817] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [10751.063164] Key type lgssc registered [10751.999237] Lustre: Echo OBD driver; http://www.lustre.org/ [10784.364874] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10784.389947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10785.666882] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10785.691079] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10785.757792] Lustre: lustre-MDT0000: new disk, initializing [10785.840104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10785.856766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10790.642057] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [10800.277023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10800.412702] Lustre: 318800:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10800.474722] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10800.482281] Lustre: Skipped 1 previous similar message [10800.550421] Lustre: lustre-MDT0001: new disk, initializing [10800.641206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10800.677136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10800.685227] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10804.690768] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [10812.876166] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10818.762971] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10825.728842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10826.317239] Lustre: lustre-OST0000: new disk, initializing [10826.329822] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10826.358737] Lustre: 320766:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10826.511485] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10833.539463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10833.562640] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10833.694546] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10833.909743] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [10845.455820] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10847.296415] Lustre: Mounted lustre-client - version 2.17.57_61_g678d580 [10848.287151] Lustre: Unmounted lustre-client [10859.487960] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10859.489260] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10859.499230] LustreError: Skipped 1 previous similar message [10859.500813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10859.526957] Lustre: Skipped 1 previous similar message [10862.566059] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10862.575822] Lustre: Skipped 2 previous similar messages [10867.683738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10867.697115] Lustre: Skipped 1 previous similar message [10872.288720] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10872.490508] Lustre: server umount lustre-OST0000 complete [10877.412106] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10877.421897] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10877.432845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10881.920809] Lustre: server umount lustre-MDT0000 complete [10885.343931] LustreError: 320765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841030 with bad export cookie 13434760976892405891 [10885.349502] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10885.354903] LustreError: 320765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10885.610655] Lustre: server umount lustre-MDT0001 complete [10894.739741] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [10897.634455] Key type lgssc unregistered [10905.119521] LNet: 322934:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10905.137167] LNetError: 322934:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10978.921109] LNet: Removed LNI 192.168.206.145@tcp [10979.812081] Key type .llcrypt unregistered [10979.813770] Key type ._llcrypt unregistered [10997.475666] Key type ._llcrypt registered [10997.478535] Key type .llcrypt registered [10997.610361] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [11013.960968] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [11015.210543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11015.241380] alg: No test for adler32 (adler32-zlib) [11016.257711] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [11016.501581] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [11018.247256] Key type lgssc registered [11019.331888] Lustre: Echo OBD driver; http://www.lustre.org/ [11049.293119] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 10:33:11 (1787841191) [11056.163979] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11056.258645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11057.594981] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11057.620223] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11057.694964] Lustre: lustre-MDT0000: new disk, initializing [11057.829061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11057.860542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11062.378984] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [11074.052427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11074.164368] Lustre: 326161:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11074.272907] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11074.285399] Lustre: Skipped 1 previous similar message [11074.421839] Lustre: lustre-MDT0001: new disk, initializing [11074.558390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11074.595475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11074.610037] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11078.555237] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [11089.749341] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11095.118349] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11102.800331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11103.065712] Lustre: lustre-OST0000: new disk, initializing [11103.070650] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11103.079289] Lustre: 328127:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11103.213649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11109.460163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11109.474524] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11109.633634] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11110.080701] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [11120.512508] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11135.461643] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11135.466033] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11135.470609] LustreError: Skipped 1 previous similar message [11135.471587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11135.491105] Lustre: Skipped 1 previous similar message [11140.577051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11140.590028] Lustre: Skipped 2 previous similar messages [11141.662887] Lustre: server umount lustre-OST0000 complete [11145.698766] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11145.709332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11145.721510] Lustre: Skipped 1 previous similar message [11150.819804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11150.822954] Lustre: Skipped 1 previous similar message [11151.747530] Lustre: server umount lustre-MDT0000 complete [11155.715424] LustreError: 328138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841300 with bad export cookie 5275725843698841712 [11155.723733] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11155.731857] LustreError: 328138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11156.006751] Lustre: server umount lustre-MDT0001 complete [11169.377730] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [11172.217135] Key type lgssc unregistered [11172.563683] LNet: 330040:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11172.579779] LNetError: 330040:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11172.621499] LNet: Removed LNI 192.168.206.145@tcp [11173.631194] Key type .llcrypt unregistered [11173.633352] Key type ._llcrypt unregistered [11197.224847] Key type ._llcrypt registered [11197.228169] Key type .llcrypt registered [11197.322454] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [11198.430835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11198.453085] alg: No test for adler32 (adler32-zlib) [11199.508668] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [11199.737328] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [11201.431327] Key type lgssc registered [11202.757942] Lustre: Echo OBD driver; http://www.lustre.org/ [11211.552540] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11211.568155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11213.168360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11218.209876] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [11222.499376] LustreError: 331315:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11222.519891] LustreError: 331315:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11225.556842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11225.899366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11229.540959] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [11236.569841] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11241.595783] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11247.236891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11247.551305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11253.230173] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [11260.084875] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11270.783431] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11272.447639] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11277.280652] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11277.292127] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11277.304815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11278.084505] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.45@tcp (stopping) [11278.312534] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11278.321956] Lustre: Skipped 1 previous similar message [11282.810563] Lustre: server umount lustre-MDT0000 complete [11283.172961] LustreError: 331312:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11286.480156] LustreError: 331297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841431 with bad export cookie 11989653471575444660 [11286.491786] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11286.499622] LustreError: 331297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11288.290689] LustreError: 331310:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11288.301211] LustreError: 331310:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11288.326372] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.45@tcp (stopping) [11288.331860] Lustre: Skipped 3 previous similar messages [11288.567430] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11291.554744] LustreError: 333329:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11291.554790] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11291.580964] LustreError: 333329:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11291.605882] Lustre: Skipped 1 previous similar message [11292.876237] Lustre: server umount lustre-MDT0001 complete [11296.638479] Lustre: server umount lustre-OST0000 complete [11305.269227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11310.189681] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [11312.884264] Lustre: 335495:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.145@tcp [11315.105337] Lustre: server umount MGS complete [11323.586663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11324.065772] LustreError: 336067:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11324.176797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11329.108908] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [11333.291106] LustreError: 336063:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11333.307946] LustreError: 336063:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11337.667149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11337.985097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11342.116928] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [11350.097155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11350.480749] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11356.616960] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [11364.197882] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11366.203701] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11371.515933] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11373.501361] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11379.168757] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11379.197801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11379.198584] Lustre: Skipped 1 previous similar message [11384.462459] Lustre: server umount lustre-OST0000 complete [11389.413629] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11389.454629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11389.460579] Lustre: Skipped 4 previous similar messages [11395.798069] Lustre: server umount lustre-MDT0000 complete [11399.648134] LustreError: 336068:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11399.677445] LustreError: 336068:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11400.219071] LustreError: 336048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841545 with bad export cookie 11989653471575445507 [11400.234407] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11400.585237] Lustre: server umount lustre-MDT0001 complete [11411.939445] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [11414.589402] Key type lgssc unregistered [11414.872645] LNet: 339505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11414.877771] LNetError: 339505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11414.908843] LNet: Removed LNI 192.168.206.145@tcp [11415.729558] Key type .llcrypt unregistered [11415.731994] Key type ._llcrypt unregistered [11435.173576] Key type ._llcrypt registered [11435.183368] Key type .llcrypt registered [11435.277961] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [11452.594648] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [11453.895157] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11453.937722] alg: No test for adler32 (adler32-zlib) [11455.037531] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [11455.364182] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [11457.127813] Key type lgssc registered [11458.118899] Lustre: Echo OBD driver; http://www.lustre.org/ [11495.183485] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11495.210632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11496.490265] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11496.570316] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11496.711417] Lustre: lustre-MDT0000: new disk, initializing [11496.835595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11496.872902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11502.234359] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [11511.876930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11512.043734] Lustre: 342665:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11512.098953] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11512.103245] Lustre: Skipped 1 previous similar message [11512.189257] Lustre: lustre-MDT0001: new disk, initializing [11512.314275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11512.412034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11512.424072] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11517.212899] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [11526.404478] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11532.471889] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11539.111431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11539.363462] Lustre: lustre-OST0000: new disk, initializing [11539.366815] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11539.371841] Lustre: 344632:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11539.455759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11545.213886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11545.234696] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11545.451796] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11546.197876] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [11556.515960] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11561.621128] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11561.840776] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11566.808246] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11567.048849] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11571.171498] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11571.175633] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11571.180641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11571.185063] Lustre: Skipped 1 previous similar message [11571.217943] LustreError: Skipped 1 previous similar message [11575.522853] Lustre: server umount lustre-OST0000 complete [11581.417382] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11581.426091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11581.443862] Lustre: Skipped 1 previous similar message [11581.465757] Lustre: Skipped 2 previous similar messages [11585.833234] Lustre: server umount lustre-MDT0000 complete [11589.794578] LustreError: 342657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841734 with bad export cookie 14444750858431952590 [11589.799079] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11589.805205] LustreError: 342657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11590.136725] Lustre: server umount lustre-MDT0001 complete [11598.813916] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 10:42:21 (1787841741) [11616.276836] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [11626.308679] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [11663.756200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11663.961467] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11663.979568] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11664.034364] Lustre: lustre-MDT0000: new disk, initializing [11664.141513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11664.167538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11668.985318] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [11680.657972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11680.815753] Lustre: 349389:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11680.882473] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11680.885639] Lustre: Skipped 1 previous similar message [11680.968205] Lustre: lustre-MDT0001: new disk, initializing [11681.096147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11681.143969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11681.162357] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11686.085517] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [11697.528350] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11703.767232] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11711.022533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11711.348414] Lustre: lustre-OST0000: new disk, initializing [11711.352819] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11711.361404] Lustre: 351360:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11711.482651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11718.893945] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [11719.254927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11719.268284] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11719.395863] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11729.117764] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11770.849147] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11770.850099] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11770.861746] LustreError: Skipped 1 previous similar message [11770.863689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11770.899969] Lustre: Skipped 1 previous similar message [11773.280555] Lustre: server umount lustre-OST0000 complete [11776.434981] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [11776.439525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11776.447222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11776.455892] Lustre: Skipped 1 previous similar message [11781.088562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11781.096871] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11782.750384] Lustre: server umount lustre-MDT0000 complete [11785.871793] LustreError: 351359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841930 with bad export cookie 14444750858431953472 [11785.879305] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11785.879601] LustreError: 351359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11786.180851] Lustre: server umount lustre-MDT0001 complete [11796.461381] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [11799.375864] Key type lgssc unregistered [11799.676752] LNet: 353535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11799.687981] LNetError: 353535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11799.710424] LNet: Removed LNI 192.168.206.145@tcp [11800.516926] Key type .llcrypt unregistered [11800.519154] Key type ._llcrypt unregistered [11895.757845] Key type ._llcrypt registered [11895.764094] Key type .llcrypt registered [11895.893400] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [11896.207499] Key type .llcrypt unregistered [11896.210765] Key type ._llcrypt unregistered [11970.348555] Key type ._llcrypt registered [11970.352037] Key type .llcrypt registered [11970.462757] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [11970.833512] Key type .llcrypt unregistered [11970.837283] Key type ._llcrypt unregistered [11997.570839] Key type ._llcrypt registered [11997.572710] Key type .llcrypt registered [11997.648417] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [11997.938398] Key type .llcrypt unregistered [11997.942422] Key type ._llcrypt unregistered [12024.930535] Key type ._llcrypt registered [12024.932188] Key type .llcrypt registered [12025.041878] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [12025.291399] Key type .llcrypt unregistered [12025.293911] Key type ._llcrypt unregistered [12095.429766] Key type ._llcrypt registered [12095.431324] Key type .llcrypt registered [12095.539169] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [12095.808346] Key type .llcrypt unregistered [12095.814469] Key type ._llcrypt unregistered [12121.156330] Key type ._llcrypt registered [12121.158794] Key type .llcrypt registered [12121.252701] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [12121.504434] Key type .llcrypt unregistered [12121.510465] Key type ._llcrypt unregistered [12148.007379] Key type ._llcrypt registered [12148.009809] Key type .llcrypt registered [12148.098568] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [12148.344468] Key type .llcrypt unregistered [12148.347172] Key type ._llcrypt unregistered [12174.821503] Key type ._llcrypt registered [12174.826636] Key type .llcrypt registered [12174.969308] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [12175.348432] Key type .llcrypt unregistered [12175.353294] Key type ._llcrypt unregistered [12203.312842] Key type ._llcrypt registered [12203.318379] Key type .llcrypt registered [12203.428621] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [12203.735320] Key type .llcrypt unregistered [12203.737870] Key type ._llcrypt unregistered [12230.424993] Key type ._llcrypt registered [12230.427361] Key type .llcrypt registered [12230.537773] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [12230.812413] Key type .llcrypt unregistered [12230.816049] Key type ._llcrypt unregistered [12290.607551] Key type ._llcrypt registered [12290.610155] Key type .llcrypt registered [12290.801192] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [12291.271308] Key type .llcrypt unregistered [12291.274035] Key type ._llcrypt unregistered [12323.438503] Key type ._llcrypt registered [12323.441029] Key type .llcrypt registered [12323.546444] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [12324.074280] Key type .llcrypt unregistered [12324.077555] Key type ._llcrypt unregistered [12358.463556] Key type ._llcrypt registered [12358.470259] Key type .llcrypt registered [12358.614676] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [12358.945271] Key type .llcrypt unregistered [12358.946510] Key type ._llcrypt unregistered [12410.701518] Key type ._llcrypt registered [12410.704692] Key type .llcrypt registered [12410.896818] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [12430.907466] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [12432.605535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12432.639259] alg: No test for adler32 (adler32-zlib) [12433.806985] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [12434.159400] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [12435.896304] Key type lgssc registered [12437.081495] Lustre: Echo OBD driver; http://www.lustre.org/ [12474.111245] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12474.140803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12475.435297] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12475.469484] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12475.587513] Lustre: tst250fs-MDT0000: new disk, initializing [12475.691789] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12475.731966] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12480.378960] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [12492.111216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12492.277293] Lustre: 373773:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12492.333425] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12492.337088] Lustre: Skipped 1 previous similar message [12492.416630] Lustre: tst250fs-MDT0001: new disk, initializing [12492.599822] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12492.640977] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12492.658365] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12496.793518] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [12501.207797] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12507.750031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12508.187914] Lustre: tst250fs-OST0000: new disk, initializing [12508.194135] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [12508.212310] Lustre: 375409:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12508.366468] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12514.441750] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12514.453663] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12514.579837] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12516.837130] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [12528.973605] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12529.177385] Lustre: tst250fs-OST0001: new disk, initializing [12529.186850] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [12529.197575] Lustre: 376281:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12529.337611] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12536.492898] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [12539.545380] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12539.560728] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12539.689261] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12541.855736] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [12544.469320] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [12544.475421] Lustre: Skipped 2 previous similar messages [12563.426997] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12563.434324] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12563.444349] Lustre: Skipped 1 previous similar message [12568.544037] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12568.563478] Lustre: Skipped 6 previous similar messages [12569.932159] Lustre: server umount tst250fs-MDT0000 complete [12574.417039] LustreError: 374572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842719 with bad export cookie 15468499224246917522 [12574.426221] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12574.435214] LustreError: 374572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12574.782489] Lustre: server umount tst250fs-MDT0001 complete [12589.376311] Lustre: server umount tst250fs-OST0000 complete [12604.372636] Lustre: server umount tst250fs-OST0001 complete [12627.819435] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [12636.278708] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [12669.303887] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 11:00:11 (1787842811) [12670.972914] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [12689.438394] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [12701.036979] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [12736.412596] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12468 sec ========= 11:01:18 (1787842878) [12738.471372] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 11:01:20 (1787842880) === [12742.089928] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 11:01:24 (1787842884) === [12781.335545] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [12783.859282] Key type lgssc unregistered [12784.066515] LNet: 385423:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12784.072197] LNetError: 385423:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12784.089644] LNet: Removed LNI 192.168.206.145@tcp [12784.779170] Key type .llcrypt unregistered [12784.781242] Key type ._llcrypt unregistered