[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 494280082 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.003202] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008445] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010011] pid_max: default: 32768 minimum: 301 [ 0.011130] LSM: Security Framework initializing [ 0.013010] Yama: becoming mindful. [ 0.014039] SELinux: Initializing. [ 0.015065] *** VALIDATE selinux *** [ 0.023755] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027798] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029088] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031033] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032110] *** VALIDATE tmpfs *** [ 0.034169] *** VALIDATE proc *** [ 0.035196] *** VALIDATE cgroup *** [ 0.036010] *** VALIDATE cgroup2 *** [ 0.038282] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039156] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041030] Spectre V2 : User space: Vulnerable [ 0.042010] Speculative Store Bypass: Vulnerable [ 0.044921] debug: unmapping init [mem 0xffffffffb1659000-0xffffffffb1660fff] [ 0.046181] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047699] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048027] ... version: 2 [ 0.049013] ... bit width: 48 [ 0.050014] ... generic registers: 4 [ 0.051012] ... value mask: 0000ffffffffffff [ 0.052015] ... max period: 00007fffffffffff [ 0.053015] ... fixed-purpose events: 3 [ 0.054030] ... event mask: 000000070000000f [ 0.055291] rcu: Hierarchical SRCU implementation. [ 0.057384] smp: Bringing up secondary CPUs ... [ 0.058549] x86: Booting SMP configuration: [ 0.059026] .... node #0, CPUs: #1 #2 #3 [ 0.062290] smp: Brought up 1 node, 4 CPUs [ 0.064011] smpboot: Max logical packages: 1 [ 0.065016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.215648] node 0 deferred pages initialised in 148ms [ 0.220112] devtmpfs: initialized [ 0.221252] x86/mm: Memory block size: 128MB [ 0.225019] gcov: version magic: 0x41383552 [ 0.227279] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.230099] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.233289] pinctrl core: initialized pinctrl subsystem [ 0.235179] [ 0.235899] ************************************************************* [ 0.239013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.241011] ** ** [ 0.244013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.246011] ** ** [ 0.248013] ** This means that this kernel is built to expose internal ** [ 0.251012] ** IOMMU data structures, which may compromise security on ** [ 0.253010] ** your system. ** [ 0.256013] ** ** [ 0.258024] ** If you see this message and you are not debugging the ** [ 0.261012] ** kernel, report this immediately to your vendor! ** [ 0.263009] ** ** [ 0.265013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.267013] ************************************************************* [ 0.269670] NET: Registered protocol family 16 [ 0.271388] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.274061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.277062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.280508] cpuidle: using governor menu [ 0.281618] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.284626] PCI: Using configuration type 1 for base access [ 0.287184] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.297099] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.299035] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.302061] cryptd: max_cpu_qlen set to 1000 [ 0.304086] ACPI: Added _OSI(Module Device) [ 0.305015] ACPI: Added _OSI(Processor Device) [ 0.307016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.308013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.314107] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.320503] ACPI: Interpreter enabled [ 0.322065] ACPI: PM: (supports S0 S3 S4 S5) [ 0.324014] ACPI: Using IOAPIC for interrupt routing [ 0.326115] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.330384] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.342419] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.344038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.348020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.351087] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.356283] acpiphp: Slot [2] registered [ 0.357139] acpiphp: Slot [5] registered [ 0.359133] acpiphp: Slot [6] registered [ 0.360094] acpiphp: Slot [7] registered [ 0.362114] acpiphp: Slot [8] registered [ 0.364133] acpiphp: Slot [9] registered [ 0.365118] acpiphp: Slot [10] registered [ 0.367100] acpiphp: Slot [3] registered [ 0.369128] acpiphp: Slot [4] registered [ 0.370107] acpiphp: Slot [11] registered [ 0.372111] acpiphp: Slot [12] registered [ 0.374130] acpiphp: Slot [13] registered [ 0.376103] acpiphp: Slot [14] registered [ 0.377103] acpiphp: Slot [15] registered [ 0.379106] acpiphp: Slot [16] registered [ 0.381117] acpiphp: Slot [17] registered [ 0.383192] acpiphp: Slot [18] registered [ 0.384117] acpiphp: Slot [19] registered [ 0.386129] acpiphp: Slot [20] registered [ 0.388108] acpiphp: Slot [21] registered [ 0.390109] acpiphp: Slot [22] registered [ 0.391110] acpiphp: Slot [23] registered [ 0.393096] acpiphp: Slot [24] registered [ 0.395125] acpiphp: Slot [25] registered [ 0.398116] acpiphp: Slot [26] registered [ 0.400153] acpiphp: Slot [27] registered [ 0.402113] acpiphp: Slot [28] registered [ 0.405126] acpiphp: Slot [29] registered [ 0.407119] acpiphp: Slot [30] registered [ 0.409116] acpiphp: Slot [31] registered [ 0.411071] PCI host bridge to bus 0000:00 [ 0.414020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.417025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.421023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.425030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.429031] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.432029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.434151] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.437983] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.440221] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.450800] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.456043] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.459024] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.461026] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.464019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.467706] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.471827] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.475049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.478894] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.485014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.498018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.504855] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.511067] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.519017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.526014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.542018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.551000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.560014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.571017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.590015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.603069] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.610014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.619014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.645017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.658512] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.664015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.669015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.685024] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.702660] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.709013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.716016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.737016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.749706] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.760013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.770014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.790013] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.804133] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.806401] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.809340] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.811354] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.814277] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.820121] iommu: Default domain type: Passthrough [ 0.822500] SCSI subsystem initialized [ 0.823119] ACPI: bus type USB registered [ 0.824115] usbcore: registered new interface driver usbfs [ 0.827075] usbcore: registered new interface driver hub [ 0.829076] usbcore: registered new device driver usb [ 0.830177] pps_core: LinuxPPS API ver. 1 registered [ 0.832012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.836059] PTP clock support registered [ 0.838077] EDAC MC: Ver: 3.0.0 [ 0.840103] PCI: Using ACPI for IRQ routing [ 0.841684] NetLabel: Initializing [ 0.843013] NetLabel: domain hash size = 128 [ 0.844010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.847091] NetLabel: unlabeled traffic allowed by default [ 0.849117] vgaarb: loaded [ 0.851253] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.852014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.859325] clocksource: Switched to clocksource kvm-clock [ 0.964427] VFS: Disk quotas dquot_6.6.0 [ 0.966142] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.968571] *** VALIDATE ramfs *** [ 0.969881] *** VALIDATE hugetlbfs *** [ 0.971634] pnp: PnP ACPI init [ 0.974043] pnp: PnP ACPI: found 6 devices [ 0.993421] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.997125] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.999624] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.002040] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.004695] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.007295] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.010382] NET: Registered protocol family 2 [ 1.012971] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.018433] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.022443] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.028376] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.032444] TCP: Hash tables configured (established 65536 bind 65536) [ 1.035203] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.037801] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.040664] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.043861] NET: Registered protocol family 1 [ 1.046546] RPC: Registered named UNIX socket transport module. [ 1.048713] RPC: Registered udp transport module. [ 1.050715] RPC: Registered tcp transport module. [ 1.052618] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.055100] NET: Registered protocol family 44 [ 1.056786] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.059117] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.061379] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.063898] PCI: CLS 0 bytes, default 64 [ 1.065614] Unpacking initramfs... [ 2.490106] debug: unmapping init [mem 0xffff8e14fcc54000-0xffff8e14fffbffff] [ 2.494560] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.496993] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.500237] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.017774] Initialise system trusted keyrings [ 3.020421] Key type blacklist registered [ 3.023955] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.036324] zbud: loaded [ 3.039929] *** VALIDATE nfs *** [ 3.041972] *** VALIDATE nfs4 *** [ 3.044417] pstore: using deflate compression [ 3.049503] Platform Keyring initialized [ 3.154891] NET: Registered protocol family 38 [ 3.156752] Key type asymmetric registered [ 3.158647] Asymmetric key parser 'x509' registered [ 3.161693] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.165920] io scheduler mq-deadline registered [ 3.168083] io scheduler kyber registered [ 3.170404] io scheduler bfq registered [ 3.172494] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.176487] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.180077] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.183223] ACPI: Power Button [PWRF] [ 3.189622] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.197599] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.212518] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.220482] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.238861] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.268028] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.295888] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.301360] Non-volatile memory driver v1.3 [ 3.303199] Linux agpgart interface v0.103 [ 3.337426] virtio_blk virtio1: [vda] 149408 512-byte logical blocks (76.5 MB/73.0 MiB) [ 3.340483] vda: detected capacity change from 0 to 76496896 [ 3.357201] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.359973] vdb: detected capacity change from 0 to 1073741824 [ 3.378511] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.382383] vdc: detected capacity change from 0 to 2621440000 [ 3.398102] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.401831] vdd: detected capacity change from 0 to 2621440000 [ 3.418965] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.422397] vde: detected capacity change from 0 to 4294967296 [ 3.436351] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.439519] vdf: detected capacity change from 0 to 4294967296 [ 3.447468] libphy: Fixed MDIO Bus: probed [ 3.456068] usbcore: registered new interface driver usbserial_generic [ 3.458633] usbserial: USB Serial support registered for generic [ 3.461305] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.465590] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.467572] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.470268] mousedev: PS/2 mouse device common for all mice [ 3.474662] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.475627] rtc_cmos 00:05: RTC can wake from S4 [ 3.481405] rtc_cmos 00:05: registered as rtc0 [ 3.481710] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.483266] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.488780] intel_pstate: CPU model not supported [ 3.490976] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.495338] hid: raw HID events driver (C) Jiri Kosina [ 3.497335] usbcore: registered new interface driver usbhid [ 3.498942] usbhid: USB HID core driver [ 3.500428] drop_monitor: Initializing network drop monitor service [ 3.502328] Initializing XFRM netlink socket [ 3.504026] NET: Registered protocol family 10 [ 3.506464] Segment Routing with IPv6 [ 3.507632] NET: Registered protocol family 17 [ 3.509250] mpls_gso: MPLS GSO support [ 3.514840] RAS: Correctable Errors collector initialized. [ 3.517230] AVX version of gcm_enc/dec engaged. [ 3.518870] AES CTR mode by8 optimization enabled [ 3.595809] sched_clock: Marking stable (3595789698, 0)->(4494842039, -899052341) [ 3.599143] registered taskstats version 1 [ 3.601288] Loading compiled-in X.509 certificates [ 3.603752] zswap: loaded using pool lzo/zbud [ 3.630324] Key type big_key registered [ 3.642492] Key type encrypted registered [ 3.644131] ima: No TPM chip found, activating TPM-bypass! [ 3.646874] ima: Allocated hash algorithm: sha1 [ 3.649130] ima: No architecture policies found [ 3.651113] evm: Initialising EVM extended attributes: [ 3.653453] evm: security.selinux [ 3.654778] evm: security.ima [ 3.656165] evm: security.capability [ 3.657582] evm: HMAC attrs: 0x1 [ 3.660153] rtc_cmos 00:05: setting system clock to 2026-08-22 13:52:54 UTC (1787406774) [ 3.667738] debug: unmapping init [mem 0xffffffffb2603000-0xffffffffb27fffff] [ 3.671433] debug: unmapping init [mem 0xffffffffb1382000-0xffffffffb1658fff] [ 3.682227] Write protecting the kernel read-only data: 28672k [ 3.685018] debug: unmapping init [mem 0xffffffffafa03000-0xffffffffafbfffff] [ 3.688188] debug: unmapping init [mem 0xffffffffb0314000-0xffffffffb03fffff] [ 3.722432] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.729801] systemd[1]: Detected virtualization kvm. [ 3.731244] systemd[1]: Detected architecture x86-64. [ 3.733295] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.755752] systemd[1]: No hostname configured. [ 3.757014] systemd[1]: Set hostname to . [ 3.758567] random: systemd: uninitialized urandom read (16 bytes read) [ 3.761268] systemd[1]: Initializing machine ID from random generator. [ 3.803806] random: ln: uninitialized urandom read (6 bytes read) [ 3.885778] random: systemd: uninitialized urandom read (16 bytes read) [ 3.888293] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.892313] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.895807] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.522144] device-mapper: uevent: version 1.0.3 [ 4.524357] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ 5.223659] random: fast init done Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.253612] virtio_net virtio0 ens2: renamed from eth0 [ 5.264912] scsi host0: ata_piix [ 5.280426] scsi host1: ata_piix [ 5.283333] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.291365] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.658446] dracut-initqueue[585]: RTNETLINK answers: File exists [ 10.162052] random: crng init done [ 10.163212] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.537040] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 13.229067] printk: systemd: 24 output lines suppressed due to ratelimiting [ 14.291566] SELinux: Disabled at runtime. [ 14.478269] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 14.504127] systemd[1]: Detected virtualization kvm. [ 14.505743] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 16.229683] systemd[1]: initrd-switch-root.service: Succeeded. [ 16.234889] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 16.260634] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 16.273690] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 16.285149] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 16.329097] systemd[1]: Starting Journal Service... Starting Journal Service... [ 16.385190] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd Root File System. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... [ OK ] Reached target Paths. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-getty.slice. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. [ 16.916761] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 17.845125] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 18.790238] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 18.823348] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 19.198078] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 19.278791] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit)[ 23.675360] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (7s / no limit)[ 24.463786] NFS: Registering the id_resolver key type [ 24.468835] Key type id_resolver registered [ 24.471774] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ 29.934895] hrtimer: interrupt took 3023562 ns [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg657-server login: [ 85.548705] libcfs: loading out-of-tree module taints kernel. [ 85.626205] Key type ._llcrypt registered [ 85.628386] Key type .llcrypt registered [ 85.732444] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 101.484590] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 102.645739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 102.661229] alg: No test for adler32 (adler32-zlib) [ 103.994564] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 104.945775] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 106.711573] Key type lgssc registered [ 108.072899] Lustre: Echo OBD driver; http://www.lustre.org/ [ 126.942642] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 161.357350] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 173.312921] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 173.347892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 174.589776] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 174.625402] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 174.743936] Lustre: lustre-MDT0000: new disk, initializing [ 174.809677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 174.835992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 178.683937] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 191.940385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 192.053254] Lustre: 6506:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 192.121894] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 192.125467] Lustre: Skipped 1 previous similar message [ 192.302284] Lustre: lustre-MDT0001: new disk, initializing [ 192.397311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 192.452018] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 192.466308] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 196.805320] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 200.962387] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 209.328628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 209.607222] Lustre: lustre-OST0000: new disk, initializing [ 209.611746] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 209.621169] Lustre: 8445:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 209.689292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 215.111053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 215.126410] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 215.176762] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 215.193502] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 228.608616] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 228.728099] Lustre: lustre-OST0001: new disk, initializing [ 228.732187] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 228.740471] Lustre: 9517:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 228.814511] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 233.906494] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 236.066089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 236.075838] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 236.140873] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 244.177206] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 249.615461] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 256.069287] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing check_logdir /tmp/testlogs/ [ 260.302745] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing yml_node [ 265.182774] Lustre: DEBUG MARKER: Client: 2.17.54.109 [ 267.782068] Lustre: DEBUG MARKER: MDS: 2.17.54.109 [ 270.419608] Lustre: DEBUG MARKER: OSS: 2.17.54.109 [ 271.899751] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 22 09:57:20 EDT 2026 [ 287.954174] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 289.166200] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 305.121805] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 305.137046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 305.155835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 307.681076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 307.684354] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 310.900253] Lustre: server umount lustre-MDT0000 complete [ 314.849268] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787407085 with bad export cookie 4470053111258319166 [ 314.855046] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 314.856448] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 317.920390] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 317.920992] LustreError: 6514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 317.924131] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 317.924140] Lustre: Skipped 3 previous similar messages [ 317.936668] Lustre: Skipped 3 previous similar messages [ 317.968478] LustreError: 6514:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 319.972571] LustreError: 6515:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 319.975558] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 320.016462] Lustre: Skipped 2 previous similar messages [ 325.090363] LustreError: 8453:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 325.091663] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 325.109471] LustreError: 8453:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 325.128474] Lustre: Skipped 1 previous similar message [ 329.183176] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 329.392489] Lustre: server umount lustre-MDT0001 complete [ 332.894577] Lustre: server umount lustre-OST0000 complete [ 335.972853] Lustre: server umount lustre-OST0001 complete [ 340.965401] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 348.742981] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 381.087723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 381.423680] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 381.470326] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 381.522900] Lustre: lustre-MDT0000: new disk, initializing [ 381.627232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 381.650972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 385.912863] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 397.788566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 397.925371] Lustre: 15068:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 397.989418] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 397.992610] Lustre: Skipped 1 previous similar message [ 398.052161] Lustre: lustre-MDT0001: new disk, initializing [ 398.115542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 398.157404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 398.167478] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 405.206337] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 415.254532] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 421.339774] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 426.990537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 427.231938] Lustre: lustre-OST0000: new disk, initializing [ 427.235624] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 427.241528] Lustre: 17038:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 427.343258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 429.134810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 429.147602] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 429.224154] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 432.941120] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 441.707853] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 445.699959] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 445.858327] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 449.999518] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 450.181399] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 455.136832] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 455.151359] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 455.162112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 457.895868] Lustre: server umount lustre-OST0000 complete [ 464.864591] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 464.873863] LustreError: Skipped 1 previous similar message [ 464.879982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 464.900838] Lustre: Skipped 1 previous similar message [ 467.293335] Lustre: server umount lustre-MDT0000 complete [ 470.498275] LustreError: 15082:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 470.526823] LustreError: 15082:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 470.613837] LustreError: 17037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787407241 with bad export cookie 4470053111258320699 [ 470.622117] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 470.626838] LustreError: 17037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 470.899695] Lustre: server umount lustre-MDT0001 complete [ 476.482150] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 10:00:45 (1787407245) [ 494.036967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 494.496375] LustreError: 19925:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 494.617761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 498.547304] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 504.803994] LustreError: 19926:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 504.840354] LustreError: 19926:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 507.925664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 512.005551] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 520.406884] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 525.366464] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 527.180991] Lustre: Failing over lustre-MDT0000 [ 527.332751] Lustre: server umount lustre-MDT0000 complete [ 528.865062] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 528.870896] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 528.881799] Lustre: Skipped 1 previous similar message [ 528.888689] LustreError: 20652:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 544.627061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 544.740841] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 547.996913] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 550.380955] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 550.386623] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 550.406680] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 556.896208] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 560.434064] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 560.435935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 560.445092] Lustre: Skipped 2 previous similar messages [ 560.456189] Lustre: Skipped 5 previous similar messages [ 566.762712] Lustre: server umount lustre-MDT0000 complete [ 570.156267] LustreError: 19906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787407340 with bad export cookie 4470053111258322113 [ 570.162870] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 570.167230] LustreError: 19906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 570.395255] Lustre: server umount lustre-MDT0001 complete [ 577.971258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 578.364745] LustreError: 23259:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 578.377080] LustreError: 23259:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 578.427070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 578.436302] Lustre: Skipped 2 previous similar messages [ 582.979656] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 591.357809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 596.651763] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 604.604177] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 609.422262] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 611.711242] Lustre: Failing over lustre-MDT0000 [ 611.956609] Lustre: server umount lustre-MDT0000 complete [ 612.323807] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 612.334105] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 629.710095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 629.862583] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 630.131460] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 634.012927] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 635.364512] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 635.372890] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 635.384092] Lustre: Skipped 1 previous similar message [ 635.412718] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 643.430478] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 650.210752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 650.215731] Lustre: Skipped 2 previous similar messages [ 653.428638] Lustre: server umount lustre-MDT0000 complete [ 655.333104] LustreError: 23259:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 655.365565] LustreError: 23259:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 657.553396] LustreError: 23240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787407428 with bad export cookie 4470053111258323051 [ 657.561087] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 657.949028] Lustre: server umount lustre-MDT0001 complete [ 668.447907] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 10:03:56 (1787407436) [ 686.803679] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 697.789591] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 740.786129] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 751.913329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 752.226246] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 752.270755] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 752.336511] Lustre: lustre-MDT0000: new disk, initializing [ 752.396803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 752.403799] Lustre: Skipped 2 previous similar messages [ 752.417740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 757.014874] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 766.274499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 766.373941] Lustre: 29635:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 766.613061] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 770.557562] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 778.543660] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 782.829456] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 788.467770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 788.769721] Lustre: 31602:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 790.331932] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 790.401942] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 790.444739] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 790.472761] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 794.373467] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 802.280632] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 806.493168] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 826.337943] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 826.338803] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 826.347373] LustreError: Skipped 1 previous similar message [ 826.361299] Lustre: Skipped 3 previous similar messages [ 826.365032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 826.377683] Lustre: Skipped 2 previous similar messages [ 832.193878] Lustre: server umount lustre-OST0000 complete [ 841.494820] Lustre: server umount lustre-MDT0000 complete [ 841.701264] LustreError: 30554:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 844.836759] LustreError: 29627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787407615 with bad export cookie 4470053111258323457 [ 844.839137] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 844.850077] LustreError: 29627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 845.166616] Lustre: server umount lustre-MDT0001 complete [ 855.822877] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 859.051874] Key type lgssc unregistered [ 859.313602] LNet: 33579:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 859.325327] LNetError: 33579:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 859.337873] LNet: Removed LNI 192.168.206.157@tcp [ 860.235173] Key type .llcrypt unregistered [ 860.240373] Key type ._llcrypt unregistered [ 882.399933] Key type ._llcrypt registered [ 882.402511] Key type .llcrypt registered [ 882.520900] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 883.282874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 883.404098] alg: No test for adler32 (adler32-zlib) [ 884.418063] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 884.622611] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 886.255163] Key type lgssc registered [ 887.245942] Lustre: Echo OBD driver; http://www.lustre.org/ [ 894.229780] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 894.256455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 895.673602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 899.471572] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 905.189229] LustreError: 34991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 905.199748] LustreError: 34991:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 906.840619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 907.096324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 910.446476] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 916.805277] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 921.619862] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 928.028692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 928.427199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 931.507815] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 934.562815] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 941.428681] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 953.410719] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 10:08:42 (1787407722) [ 954.990697] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 956.451363] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 962.666698] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 10:08:51 (1787407731) [ 970.182470] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 10:08:58 (1787407738) [ 971.270328] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 980.820568] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 980.826476] Lustre: Skipped 2 previous similar messages [ 982.525402] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 10:09:11 (1787407751) [ 986.272731] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 986.280371] Lustre: Skipped 3 previous similar messages [ 1004.101091] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1004.110659] Lustre: Skipped 6 previous similar messages [ 1006.638892] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 10:09:35 (1787407775) [ 1009.133437] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1012.413168] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1019.893906] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 10:09:48 (1787407788) [ 1021.573703] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1021.578907] Lustre: Skipped 3 previous similar messages [ 1033.926623] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 10:10:02 (1787407802) [ 1048.560831] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1048.576511] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1048.579879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1048.597086] Lustre: Skipped 1 previous similar message [ 1048.618562] Lustre: Skipped 1 previous similar message [ 1053.666053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1053.672656] Lustre: Skipped 1 previous similar message [ 1058.785466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1058.795571] Lustre: Skipped 1 previous similar message [ 1062.367109] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1062.481045] Lustre: server umount lustre-OST0000 complete [ 1069.033681] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1069.042389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1069.048568] Lustre: Skipped 1 previous similar message [ 1069.069090] Lustre: Skipped 1 previous similar message [ 1072.494359] Lustre: server umount lustre-MDT0000 complete [ 1074.145299] LustreError: 34992:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1074.164365] LustreError: 34992:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1076.677785] LustreError: 38551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787407847 with bad export cookie 2295454485137515812 [ 1076.679712] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1076.692245] LustreError: 38551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1077.212143] Lustre: server umount lustre-MDT0001 complete [ 1088.421474] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 1091.009836] Key type lgssc unregistered [ 1091.426948] LNet: 42852:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1091.431786] LNetError: 42852:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1091.464817] LNet: Removed LNI 192.168.206.157@tcp [ 1092.437127] Key type .llcrypt unregistered [ 1092.440792] Key type ._llcrypt unregistered [ 1109.606984] Key type ._llcrypt registered [ 1109.610202] Key type .llcrypt registered [ 1109.739613] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 1124.112951] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 1124.914648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1124.977106] alg: No test for adler32 (adler32-zlib) [ 1125.972597] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 1126.174410] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 1127.807143] Key type lgssc registered [ 1128.693830] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1159.338983] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1159.417179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1160.737126] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1160.762906] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1160.839683] Lustre: lustre-MDT0000: new disk, initializing [ 1160.936655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1160.952622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1165.848910] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1176.204434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1176.348092] Lustre: 46010:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1176.411505] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1176.414862] Lustre: Skipped 1 previous similar message [ 1176.473834] Lustre: lustre-MDT0001: new disk, initializing [ 1176.555578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1176.599633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1176.606891] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1180.336763] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1189.258736] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1194.808317] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1201.579651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1201.868876] Lustre: lustre-OST0000: new disk, initializing [ 1201.874391] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1201.881317] Lustre: 47978:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1202.009875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1208.192731] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1211.967215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1211.978907] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1212.121516] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1217.524391] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1221.804393] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1221.981820] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1225.615738] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1225.835793] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1227.748792] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1227.749230] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1227.757188] LustreError: Skipped 1 previous similar message [ 1227.770303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1227.780180] Lustre: Skipped 1 previous similar message [ 1232.868056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1232.871754] Lustre: Skipped 1 previous similar message [ 1233.598850] Lustre: server umount lustre-OST0000 complete [ 1237.985277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1237.988774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1238.000129] Lustre: Skipped 2 previous similar messages [ 1238.019590] Lustre: Skipped 1 previous similar message [ 1243.105196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1243.112962] Lustre: Skipped 1 previous similar message [ 1248.226771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1248.236165] Lustre: Skipped 1 previous similar message [ 1251.295160] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1251.454100] Lustre: server umount lustre-MDT0000 complete [ 1255.467524] LustreError: 46001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787408026 with bad export cookie 388241425826036046 [ 1255.480975] LustreError: 46001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1255.486532] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1255.824864] Lustre: server umount lustre-MDT0001 complete [ 1265.613543] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 10:13:54 (1787408034) [ 1273.212655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1274.054578] LustreError: 50441:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1274.074557] LustreError: 50441:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1274.149359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1279.458568] LustreError: 50442:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1280.346983] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1283.554781] LustreError: 50441:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1288.691767] LustreError: 50442:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1289.459797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1289.872227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1294.207831] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1301.463333] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1306.839740] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1312.770261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1313.113568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1319.565331] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1326.281118] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1330.573716] LustreError: 52971:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1331.527414] Lustre: Modifying parameter general.timeout=1 in log params [ 1333.304810] Lustre: Modifying parameter general.timeout=3 in log params [ 1333.309368] Lustre: Skipped 1 previous similar message [ 1335.909283] Lustre: Modifying parameter general.timeout=6 in log params [ 1335.915285] Lustre: Skipped 2 previous similar messages [ 1340.633702] Lustre: Modifying parameter general.timeout=12 in log params [ 1340.639565] Lustre: Skipped 5 previous similar messages [ 1348.663847] Lustre: Modifying parameter general.timeout=21 in log params [ 1348.670252] Lustre: Skipped 8 previous similar messages [ 1364.845080] Lustre: Modifying parameter general.timeout=41 in log params [ 1364.849131] Lustre: Skipped 19 previous similar messages [ 1397.281307] Lustre: Modifying parameter general.timeout=81 in log params [ 1397.292991] Lustre: Skipped 39 previous similar messages [ 1452.413383] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 10:17:01 (1787408221) [ 1455.383635] Lustre: 59912:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1457.318835] Lustre: 60010:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1459.508913] Lustre: 60122:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1462.443570] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 in log params [ 1462.451642] Lustre: Skipped 54 previous similar messages [ 1465.301274] Lustre: 60417:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1465.318225] Lustre: 60417:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1489.722776] Lustre: 61776:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1489.766156] LustreError: 61776:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1493.294665] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 10:17:41 (1787408261) [ 1507.298199] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1507.302686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1507.310611] Lustre: Skipped 2 previous similar messages [ 1507.315458] Lustre: Skipped 1 previous similar message [ 1511.691622] Lustre: server umount lustre-MDT0000 complete [ 1512.417239] LustreError: 50441:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1512.434379] LustreError: 50441:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1515.065767] LustreError: 50423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787408285 with bad export cookie 388241425826036928 [ 1515.067844] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1515.077171] LustreError: 50423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1515.476683] Lustre: server umount lustre-MDT0001 complete [ 1528.450385] Lustre: server umount lustre-OST0000 complete [ 1554.892719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1555.068696] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1555.328147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1559.340327] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1566.512630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1566.655474] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1566.683238] Lustre: 64309:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1571.130966] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1579.233142] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1585.471605] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1592.030011] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1592.176708] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1592.437360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1592.457557] Lustre: Skipped 1 previous similar message [ 1597.732444] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1603.377512] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1604.756991] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1604.766710] Lustre: Skipped 17 previous similar messages [ 1604.805592] Lustre: 66873:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1613.793293] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1613.794072] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1613.805738] LustreError: Skipped 1 previous similar message [ 1613.827586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1613.834872] Lustre: Skipped 2 previous similar messages [ 1617.201790] Lustre: server umount lustre-OST0000 complete [ 1623.520687] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1623.533103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1623.554049] Lustre: Skipped 1 previous similar message [ 1626.830518] Lustre: server umount lustre-MDT0000 complete [ 1629.153197] LustreError: 64319:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1629.170112] LustreError: 64319:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1630.222269] LustreError: 64300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787408401 with bad export cookie 388241425826039931 [ 1630.224166] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1630.243870] LustreError: 64300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1630.480768] Lustre: server umount lustre-MDT0001 complete [ 1639.691190] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 1641.858650] Key type lgssc unregistered [ 1642.073315] LNet: 68081:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1642.078857] LNetError: 68081:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1642.099745] LNet: Removed LNI 192.168.206.157@tcp [ 1642.865469] Key type .llcrypt unregistered [ 1642.875201] Key type ._llcrypt unregistered [ 1665.496356] Key type ._llcrypt registered [ 1665.499466] Key type .llcrypt registered [ 1665.634842] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 1666.871566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1666.900394] alg: No test for adler32 (adler32-zlib) [ 1667.968928] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 1668.206446] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 1669.863164] Key type lgssc registered [ 1670.859648] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1677.771057] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1677.785929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1679.123354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1682.923839] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1688.545933] LustreError: 69493:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1688.561968] LustreError: 69493:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1690.260129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1690.752785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1694.092704] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1700.272931] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1704.738654] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1710.679324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1711.233052] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1717.215191] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1723.361768] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1730.355268] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1739.233905] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1739.236424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1739.248723] Lustre: Skipped 2 previous similar messages [ 1739.262171] Lustre: Skipped 2 previous similar messages [ 1743.620107] Lustre: server umount lustre-MDT0000 complete [ 1744.355640] LustreError: 69492:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1744.373546] LustreError: 69492:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1746.546761] LustreError: 72172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787408517 with bad export cookie 3029116800991120130 [ 1746.552575] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1746.561834] LustreError: 72172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1746.780490] Lustre: server umount lustre-MDT0001 complete [ 1760.620558] Lustre: server umount lustre-OST0000 complete [ 1784.449308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1784.707460] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1784.739366] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1784.981889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1788.497939] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1798.643801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1798.746370] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1798.765910] Lustre: 74362:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1799.249943] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1804.853854] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1812.087966] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1817.170524] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1824.002930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1824.140938] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1824.513225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1829.901713] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1836.062758] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1837.576897] Lustre: 76935:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1839.448423] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1846.244036] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1846.250750] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1846.263483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1846.765271] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1851.616444] Lustre: server umount lustre-OST0000 complete [ 1855.465887] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1855.474594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1855.485740] Lustre: Skipped 1 previous similar message [ 1855.488152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1855.491873] Lustre: Skipped 1 previous similar message [ 1862.113105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1862.124873] Lustre: Skipped 3 previous similar messages [ 1866.216653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1866.221149] Lustre: Skipped 1 previous similar message [ 1869.279126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1869.364666] Lustre: server umount lustre-MDT0000 complete [ 1871.334626] LustreError: 74375:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1871.353069] LustreError: 74375:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1872.683836] LustreError: 74354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787408643 with bad export cookie 3029116800991120970 [ 1872.687971] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1872.701457] LustreError: 74354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1872.886936] Lustre: server umount lustre-MDT0001 complete [ 1882.883469] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 1885.071163] Key type lgssc unregistered [ 1885.342889] LNet: 78153:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1885.353297] LNetError: 78153:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1885.369092] LNet: Removed LNI 192.168.206.157@tcp [ 1886.108507] Key type .llcrypt unregistered [ 1886.111290] Key type ._llcrypt unregistered [ 1910.924506] Key type ._llcrypt registered [ 1910.926720] Key type .llcrypt registered [ 1911.093991] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 1912.133072] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1912.145150] alg: No test for adler32 (adler32-zlib) [ 1913.179028] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 1913.403051] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 1915.063309] Key type lgssc registered [ 1916.035597] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1923.195913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1923.210499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1924.672499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1928.929721] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1934.819498] LustreError: 79568:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1934.851489] LustreError: 79568:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1936.623791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1937.276232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1940.867683] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1946.944174] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1951.553606] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1957.234788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1957.817688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1962.565690] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1968.282338] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1978.890150] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1979.746893] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1981.375388] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1981.380342] Lustre: Skipped 1 previous similar message [ 1983.945403] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1983.950067] Lustre: Skipped 2 previous similar messages [ 1987.954277] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1987.962060] Lustre: Skipped 3 previous similar messages [ 1996.130836] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1996.140551] Lustre: Skipped 8 previous similar messages [ 2012.492886] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2012.497659] Lustre: Skipped 17 previous similar messages [ 2045.349185] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2045.352538] Lustre: Skipped 39 previous similar messages [ 2070.242567] Lustre: 87661:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2072.646962] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 10:27:21 (1787408841) [ 2074.108189] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2075.869312] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 10:27:24 (1787408844) [ 2079.714862] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2079.722708] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2079.742416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2080.738481] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2080.741716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2080.758267] Lustre: Skipped 1 previous similar message [ 2085.445768] Lustre: server umount lustre-OST0000 complete [ 2090.464644] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2090.469931] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2090.492165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2094.880924] Lustre: server umount lustre-MDT0000 complete [ 2096.103162] LustreError: 80299:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2096.123196] LustreError: 80299:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2098.011392] LustreError: 80308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787408868 with bad export cookie 17796745125982665235 [ 2098.013948] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2098.031145] LustreError: 80308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2098.304134] Lustre: server umount lustre-MDT0001 complete [ 2107.711921] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 2110.126241] Key type lgssc unregistered [ 2110.456995] LNet: 88910:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2110.467746] LNetError: 88910:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2110.483091] LNet: Removed LNI 192.168.206.157@tcp [ 2111.227392] Key type .llcrypt unregistered [ 2111.229754] Key type ._llcrypt unregistered [ 2114.838292] Key type ._llcrypt registered [ 2114.843377] Key type .llcrypt registered [ 2114.948272] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2118.737137] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules [ 2119.797268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2119.841235] alg: No test for adler32 (adler32-zlib) [ 2120.856728] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 2121.074894] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 2122.767202] Key type lgssc registered [ 2123.858432] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2125.191704] LustreError: 90111:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2141.840582] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 2147.026079] LustreError: 90111:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2150.411500] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2150.440931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2151.873054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2155.745782] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2162.148179] LustreError: 91134:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2162.163721] LustreError: 91134:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2163.884366] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 10:28:52 (1787408932) [ 2167.279771] LustreError: 91135:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2168.032685] LustreError: 91166:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2168.044288] LustreError: 91166:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2168.061564] LustreError: 91166:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 0, failed: rc = -5 [ 2168.440785] Lustre: server umount lustre-MDT0000 complete [ 2180.602519] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 2183.065527] Key type lgssc unregistered [ 2183.320763] LNet: 92659:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2183.331408] LNetError: 92659:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2183.351819] LNet: Removed LNI 192.168.206.157@tcp [ 2184.075975] Key type .llcrypt unregistered [ 2184.079117] Key type ._llcrypt unregistered [ 2199.243874] Key type ._llcrypt registered [ 2199.246030] Key type .llcrypt registered [ 2199.325791] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 2200.119326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2200.238542] alg: No test for adler32 (adler32-zlib) [ 2201.253375] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 2201.471984] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 2203.159174] Key type lgssc registered [ 2204.404358] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2211.679275] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2211.698194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2213.265618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2217.342925] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2222.562446] LustreError: 93871:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2222.572489] LustreError: 93871:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2225.567415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2226.113891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2230.090857] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2237.306996] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2243.104888] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2250.017374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2250.648093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2256.444434] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2264.245476] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2311.199958] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 10:31:19 (1787409079) [ 2312.844067] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2314.421114] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 10:31:23 (1787409083) [ 2318.307915] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2318.316825] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2318.333094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2319.330678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2319.331216] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2319.334863] Lustre: Skipped 1 previous similar message [ 2319.351775] Lustre: Skipped 1 previous similar message [ 2324.278598] Lustre: server umount lustre-MDT0000 complete [ 2324.456784] LustreError: 93871:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2324.477455] LustreError: 93871:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2327.457834] LustreError: 93853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787409098 with bad export cookie 5876443577519773894 [ 2327.465026] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2327.470575] LustreError: 93853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2327.711384] Lustre: server umount lustre-MDT0001 complete [ 2340.374718] Lustre: server umount lustre-OST0000 complete [ 2350.879686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2351.287598] LustreError: 98163:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2351.470707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2355.335307] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2356.706072] LustreError: 98164:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2361.824637] LustreError: 98163:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2362.665476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2363.255658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2367.133361] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2373.456369] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2377.741735] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2388.746914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2388.929506] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2388.941128] LustreError: 98153:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2388.966674] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2388.974439] LustreError: 100245:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2388.988186] LustreError: 100245:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2388.993685] LustreError: 100245:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2389.048921] Lustre: server umount lustre-OST0000 complete [ 2389.052421] LustreError: 100245:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2395.625955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2395.776699] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2395.781834] LustreError: 98153:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2395.789819] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2395.797988] LustreError: 100708:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2395.805458] LustreError: 100708:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2395.810451] LustreError: 100708:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2395.863434] Lustre: server umount lustre-OST0000 complete [ 2395.866664] LustreError: 100708:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2404.038356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2404.228358] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2404.235496] Lustre: Found index 0 for lustre-OST0000, updating log [ 2404.242840] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2404.337893] Lustre: lustre-OST0000: new disk, initializing [ 2404.353421] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2404.364334] Lustre: 101242:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2404.579724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2409.960790] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2413.283729] Lustre: Failing over lustre-OST0000 [ 2413.391263] Lustre: server umount lustre-OST0000 complete [ 2418.144806] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2418.153808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2418.164944] Lustre: Skipped 1 previous similar message [ 2418.168444] Lustre: Skipped 1 previous similar message [ 2422.747644] Lustre: server umount lustre-MDT0000 complete [ 2423.269908] LustreError: 98893:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2423.291831] LustreError: 98893:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2425.909706] LustreError: 98145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787409196 with bad export cookie 5876443577519776638 [ 2425.914138] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2425.916949] LustreError: 98145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2436.180438] Lustre: server umount lustre-MDT0001 complete [ 2443.070228] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 10:33:31 (1787409211) [ 2455.531813] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 2462.846166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2463.137392] LustreError: 103584:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2463.229995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2466.089161] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2471.832484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2475.267187] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2477.695859] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2483.386574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2483.997423] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2489.137359] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2489.142736] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2489.158713] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -16 [ 2489.465636] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2494.437320] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:55 [ 2494.454130] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -16 [ 2496.499328] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:53 [ 2496.526557] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -16 [ 2496.547971] Lustre: lustre-OST0000: Recovery over after 0:07, of 1 clients 1 recovered and 0 were evicted. [ 2496.557511] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2497.179648] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2497.244808] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2497.325722] Lustre: lustre-OST0001: new disk, initializing [ 2497.329431] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2497.338023] Lustre: 105902:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2497.588545] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2497.599956] Lustre: Skipped 2 previous similar messages [ 2501.886191] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2506.792552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2506.806082] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2506.868805] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2507.706674] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2514.184155] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2519.182574] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.57@tcp (stopping) [ 2519.186703] Lustre: Skipped 1 previous similar message [ 2519.520531] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2519.528029] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2523.579000] Lustre: server umount lustre-MDT0001 complete [ 2524.257309] LustreError: 103579:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2524.276092] LustreError: 103579:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2531.119928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2531.183978] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2531.187711] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2531.220743] Lustre: 103573:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2532.923919] LustreError: 107491:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2532.933705] LustreError: 107491:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2532.940876] LustreError: 103576:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2533.022797] LustreError: 107501:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.206.157@tcp: cfg command failed: rc = -17 [ 2533.037811] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2533.037811] [ 2535.386351] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2536.953210] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2543.894608] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 10:35:12 (1787409312) [ 2558.481725] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 2571.336314] Lustre: 109517:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2587.878981] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2590.481862] Lustre: 110651:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2664.416718] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2664.426061] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2664.434862] Lustre: Skipped 2 previous similar messages [ 2664.437852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2664.441718] Lustre: Skipped 4 previous similar messages [ 2670.050741] LustreError: 103579:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2670.078469] LustreError: 103579:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2670.136715] Lustre: server umount lustre-MDT0000 complete [ 2672.831546] LustreError: 103565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787409443 with bad export cookie 5876443577519777562 [ 2672.836894] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2672.844807] LustreError: 103565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2673.097492] Lustre: server umount lustre-MDT0001 complete [ 2686.238654] Lustre: server umount lustre-OST0000 complete [ 2698.355798] Lustre: server umount lustre-OST0001 complete [ 2702.001980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2707.837624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2712.606557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2719.179334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2731.714736] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 2738.638912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2738.658491] Lustre: lustre-MDT0000: reset Object Index mappings [ 2739.020159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2739.028733] Lustre: Skipped 1 previous similar message [ 2742.293498] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2748.456206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2748.477973] Lustre: lustre-MDT0001: reset Object Index mappings [ 2752.041933] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2754.033311] Lustre: 115747:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2758.526429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2759.981078] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:226 to 0x280000400:257) [ 2763.639710] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2764.085175] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 2770.129443] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2775.217962] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2776.051324] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 2776.051540] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 2781.518050] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2784.284748] Lustre: 117623:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2823.115950] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 10:39:52 (1787409592) [ 2851.296260] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2851.310572] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2851.326163] Lustre: Skipped 3 previous similar messages [ 2851.331991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2851.338588] Lustre: Skipped 3 previous similar messages [ 2857.063611] Lustre: server umount lustre-MDT0000 complete [ 2857.954752] LustreError: 114601:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2857.970128] LustreError: 114601:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 2860.162985] LustreError: 114582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787409631 with bad export cookie 5876443577519824448 [ 2860.171449] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2860.173498] LustreError: 114582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2866.533897] Lustre: server umount lustre-MDT0001 complete [ 2875.645722] Lustre: server umount lustre-OST0000 complete [ 2884.433688] Lustre: server umount lustre-OST0001 complete [ 2888.740917] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 2895.401919] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 2918.598193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2918.775404] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2918.795366] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2918.863476] Lustre: lustre-MDT0000: new disk, initializing [ 2918.919685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2918.929977] Lustre: Skipped 3 previous similar messages [ 2918.943503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2922.005854] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2926.877732] Lustre: server umount lustre-MDT0000 complete [ 2932.871867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2933.099395] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2933.115438] Lustre: 121897:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2933.129732] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2936.076414] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2939.238817] Lustre: server umount lustre-MDT0000 complete [ 2945.477584] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 10:41:54 (1787409714) [ 2946.587944] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2947.849460] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 10:41:56 (1787409716) [ 2960.766443] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 2967.383841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2970.623910] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2976.290152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2976.372962] Lustre: 123947:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2976.415969] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2976.419548] Lustre: Skipped 1 previous similar message [ 2976.479585] Lustre: lustre-MDT0001: new disk, initializing [ 2976.567351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2976.573623] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2979.375855] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2982.810079] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2987.319038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2987.565106] Lustre: lustre-OST0000: new disk, initializing [ 2987.569602] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2987.577996] Lustre: 125522:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2989.157743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2989.166860] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2989.233954] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2991.641030] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2999.183223] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2999.308630] Lustre: lustre-OST0001: new disk, initializing [ 2999.312358] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2999.317898] Lustre: 126392:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3000.753787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3000.762453] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3000.843329] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3003.305590] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3010.333485] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3013.150704] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3264.093091] Lustre: 132787:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3264.099557] Lustre: 132787:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3264.106119] Lustre: 132787:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3264.110491] Lustre: 132787:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3264.119516] Lustre: 132787:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3264.124429] Lustre: 132787:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3430.329685] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:49:59 (1787410199) [ 3442.656372] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3442.659776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3442.666532] Lustre: Skipped 5 previous similar messages [ 3442.669395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3442.672433] Lustre: Skipped 8 previous similar messages [ 3444.903391] Lustre: server umount lustre-MDT0000 complete [ 3446.811355] LustreError: 123939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787410217 with bad export cookie 5876443577519889254 [ 3446.817521] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3446.818964] LustreError: 123939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3449.314899] LustreError: 123952:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3449.328415] LustreError: 123952:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3453.143818] Lustre: server umount lustre-MDT0001 complete [ 3469.279190] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3469.337407] Lustre: server umount lustre-OST0000 complete [ 3477.333091] Lustre: server umount lustre-OST0001 complete [ 3480.356619] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 3485.194953] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 3500.934840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3501.050349] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3501.070618] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3501.113384] Lustre: lustre-MDT0000: new disk, initializing [ 3501.159168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3501.163200] Lustre: Skipped 5 previous similar messages [ 3501.171633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3503.118357] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3509.015323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3509.066381] Lustre: 135814:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3509.246249] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3511.350041] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3516.642954] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3519.605265] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3523.312602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3523.462500] Lustre: 137782:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3524.570978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3524.573791] Lustre: Skipped 1 previous similar message [ 3524.576360] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3524.632818] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3526.374351] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3531.435272] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3536.296798] Lustre: lustre-MDD0000: changelog on [ 3537.678249] Lustre: lustre-MDD0001: changelog on [ 4289.063240] Lustre: lustre-MDD0001: changelog off [ 4289.779824] Lustre: 135822:0:(llog_cat.c:984:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 4290.130451] Lustre: lustre-MDD0000: changelog off [ 4292.577952] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4292.579399] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4292.581485] LustreError: Skipped 1 previous similar message [ 4292.585837] Lustre: Skipped 5 previous similar messages [ 4292.588842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4292.592078] Lustre: Skipped 9 previous similar messages [ 4297.698567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4297.701048] Lustre: Skipped 1 previous similar message [ 4306.399150] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4306.463168] Lustre: server umount lustre-OST0000 complete [ 4307.936642] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4307.936711] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4307.937392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4307.937396] Lustre: Skipped 2 previous similar messages [ 4307.948478] Lustre: Skipped 1 previous similar message [ 4322.271312] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4322.360602] Lustre: server umount lustre-MDT0000 complete [ 4323.295842] LustreError: 136618:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4323.302941] LustreError: 136618:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4323.644901] LustreError: 135808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787411094 with bad export cookie 5876443577520704425 [ 4323.648987] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4323.650180] LustreError: 135808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4323.761940] Lustre: server umount lustre-MDT0001 complete [ 4327.865260] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 4328.904744] Key type lgssc unregistered [ 4329.044373] LNet: 146718:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4329.048090] LNetError: 146718:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4329.057404] LNet: Removed LNI 192.168.206.157@tcp [ 4329.403156] Key type .llcrypt unregistered [ 4329.404549] Key type ._llcrypt unregistered [ 4336.443200] Key type ._llcrypt registered [ 4336.444118] Key type .llcrypt registered [ 4336.483049] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 4343.119490] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 4343.545450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4343.557431] alg: No test for adler32 (adler32-zlib) [ 4344.431635] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 4344.530910] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 4346.119150] Key type lgssc registered [ 4346.526710] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4358.536799] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4358.547426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4359.632559] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4359.643995] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4359.683579] Lustre: lustre-MDT0000: new disk, initializing [ 4359.714548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4359.723066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4361.135318] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4365.500314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4365.547193] Lustre: 149891:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4365.582702] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4365.584365] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4365.587627] Lustre: Skipped 1 previous similar message [ 4365.621864] Lustre: lustre-MDT0001: new disk, initializing [ 4365.657072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4365.672357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4365.676321] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4367.193394] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4371.372781] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4373.414475] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4376.084356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4376.215211] Lustre: lustre-OST0000: new disk, initializing [ 4376.217448] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4376.221814] Lustre: 151856:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4376.258046] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4378.666726] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4381.178521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4381.182702] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4381.228059] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4382.971482] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4386.472529] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_module obdecho/obdecho [ 4387.230872] LustreError: 152860:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 4387.235619] LustreError: 152860:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 4387.958944] LustreError: 152956:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 4396.515792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4396.517123] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4396.517717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4396.519456] Lustre: Skipped 2 previous similar messages [ 4398.048743] Lustre: server umount lustre-MDT0000 complete [ 4399.483331] LustreError: 150685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787411170 with bad export cookie 809840706254347118 [ 4399.485869] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4399.489279] LustreError: 150685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4399.610257] Lustre: server umount lustre-MDT0001 complete [ 4411.424698] Lustre: server umount lustre-OST0000 complete [ 4419.924091] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 4424.514593] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 4440.371130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4440.481378] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4440.497213] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4440.536920] Lustre: lustre-MDT0000: new disk, initializing [ 4440.564709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4440.573357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4442.237708] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4447.026527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4447.078035] Lustre: 156666:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4447.111600] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4447.114350] Lustre: Skipped 1 previous similar message [ 4447.156757] Lustre: lustre-MDT0001: new disk, initializing [ 4447.202977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4447.219252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4447.224162] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4448.901334] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4453.556640] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4456.065346] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4459.140758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4459.271445] Lustre: lustre-OST0000: new disk, initializing [ 4459.274785] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4459.280434] Lustre: 158635:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4459.318263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4461.649826] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4463.297730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4463.301825] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4463.338410] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4466.009688] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4467.883289] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4467.984308] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4469.680564] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4469.774528] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4473.826411] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4473.826836] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4473.829880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4473.829885] Lustre: Skipped 2 previous similar messages [ 4473.830316] LustreError: Skipped 1 previous similar message [ 4473.836278] Lustre: Skipped 1 previous similar message [ 4476.634653] Lustre: server umount lustre-OST0000 complete [ 4478.944756] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4478.945482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4478.950465] Lustre: Skipped 1 previous similar message [ 4478.953669] Lustre: Skipped 1 previous similar message [ 4484.064117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4484.067350] Lustre: Skipped 2 previous similar messages [ 4484.250200] Lustre: server umount lustre-MDT0000 complete [ 4485.670238] LustreError: 158634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787411256 with bad export cookie 809840706254348077 [ 4485.672733] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4485.675812] LustreError: 158634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4485.799359] Lustre: server umount lustre-MDT0001 complete [ 4487.124305] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 11:07:36 (1787411256) [ 4487.750038] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 4488.476573] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 11:07:37 (1787411257) [ 4491.397182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4491.630596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4493.308245] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4496.842908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4496.864628] LustreError: 161044:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4496.875182] LustreError: 161044:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4498.714589] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4501.811700] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4504.122746] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4507.006702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4509.634494] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4512.619228] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4526.562213] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4526.564742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4526.567625] Lustre: Skipped 2 previous similar messages [ 4526.570033] Lustre: Skipped 1 previous similar message [ 4529.250554] Lustre: server umount lustre-MDT0000 complete [ 4530.863534] LustreError: 161024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787411301 with bad export cookie 809840706254348959 [ 4530.867711] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4530.870446] LustreError: 161024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4531.680090] LustreError: 161040:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4531.683120] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4533.343896] LustreError: 161038:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4535.712469] LustreError: 161039:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4537.109917] Lustre: server umount lustre-MDT0001 complete [ 4539.933730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4545.023911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4545.123581] LustreError: 164901:0:(llog.c:1655:llog_backup()) MGC192.168.206.157@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 4545.128043] Lustre: 164901:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.157@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 4553.822828] LustreError: 164906:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4564.062845] LustreError: 164906:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4564.068856] LustreError: 164906:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4580.831971] LustreError: 164907:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4580.838653] LustreError: 164907:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 4606.431284] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 4606.434121] Lustre: 164901:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.206.157@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 4615.262668] LustreError: 164925:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4615.269583] LustreError: 164925:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 21 previous similar messages [ 4646.885200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4646.889141] Lustre: Skipped 2 previous similar messages [ 4648.456119] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4650.981195] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4651.510037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4675.551826] LustreError: 148118:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006e59a416 x1874236397426944/t0(0) o250->MGC192.168.206.157@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4677.160792] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4679.787902] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4680.349996] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4680.681344] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4682.155389] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4685.728432] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 3 sec [ 4690.914216] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4690.914303] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4690.916022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4690.916032] Lustre: Skipped 5 previous similar messages [ 4690.927932] Lustre: Skipped 2 previous similar messages [ 4692.828689] Lustre: server umount lustre-MDT0000 complete [ 4694.197569] LustreError: 161023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787411465 with bad export cookie 809840706254355406 [ 4694.201116] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4694.202026] LustreError: 161023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4694.325193] Lustre: server umount lustre-MDT0001 complete [ 4706.019760] Lustre: server umount lustre-OST0000 complete [ 4713.716275] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 4717.285835] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 4730.202775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4730.306249] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4730.319793] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4730.358216] Lustre: lustre-MDT0000: new disk, initializing [ 4730.383410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4730.386372] Lustre: Skipped 1 previous similar message [ 4730.392691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4731.808952] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4736.041892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4736.081776] Lustre: 169972:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4736.190732] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4737.517191] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4741.207872] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4743.063783] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4745.329654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4745.451095] Lustre: 171941:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4747.485698] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4751.152881] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4752.652248] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4755.445619] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4755.477034] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4755.787080] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 4757.234678] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4757.314271] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4760.546432] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4760.548389] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4760.553174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4760.556053] Lustre: Skipped 3 previous similar messages [ 4764.047843] Lustre: server umount lustre-OST0000 complete [ 4771.422589] Lustre: server umount lustre-MDT0000 complete [ 4772.672297] LustreError: 169966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787411543 with bad export cookie 809840706254372234 [ 4772.675639] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4772.676591] LustreError: 169966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4772.786415] Lustre: server umount lustre-MDT0001 complete [ 4775.615426] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 11:12:25 (1787411545) [ 4777.840308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4778.006377] LustreError: 174370:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4778.013109] LustreError: 174370:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 45 previous similar messages [ 4779.366362] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4782.240949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4783.745825] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4786.108648] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4787.906644] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4790.146042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4792.250111] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4794.703181] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4809.140203] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 11:12:58 (1787411578) [ 4812.770247] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4812.770669] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4812.774531] Lustre: Skipped 4 previous similar messages [ 4812.776827] LustreError: Skipped 1 previous similar message [ 4816.215955] Lustre: server umount lustre-OST0000 complete [ 4823.707378] Lustre: server umount lustre-MDT0000 complete [ 4825.006105] LustreError: 174351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787411595 with bad export cookie 809840706254373102 [ 4825.009949] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4825.010466] LustreError: 174351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4825.136662] Lustre: server umount lustre-MDT0001 complete [ 4829.010840] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 4830.051410] Key type lgssc unregistered [ 4830.186338] LNet: 178118:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4830.188650] LNetError: 178118:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4830.198287] LNet: Removed LNI 192.168.206.157@tcp [ 4830.502311] Key type .llcrypt unregistered [ 4830.503452] Key type ._llcrypt unregistered [ 4838.217524] Key type ._llcrypt registered [ 4838.219100] Key type .llcrypt registered [ 4838.271173] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 4838.702747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4838.741496] alg: No test for adler32 (adler32-zlib) [ 4839.619120] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 4839.730987] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 4841.319155] Key type lgssc registered [ 4841.698229] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4844.450381] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4844.457866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4859.871447] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4864.991320] LustreError: 179350:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.157@tcp: failed processing log, type 4: rc = -110 [ 4870.111523] LustreError: 179350:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4870.114907] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 4870.116540] LustreError: MGC192.168.206.157@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 4870.119103] LustreError: 179350:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 4870.123201] LustreError: 179350:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 4870.126786] LustreError: 179350:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 4870.146595] Lustre: server umount lustre-OST0000 complete [ 4870.148025] LustreError: 179350:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4873.107769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4874.259674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4875.606083] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4878.579447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4878.732964] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4880.264595] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4882.812110] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4884.669080] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4887.149078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4887.277510] LustreError: 181661:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4887.281055] Lustre: 181661:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.206.157@tcp: invalid local config log lustre-OST0000: rc = -22 [ 4887.334901] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4889.417521] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4891.936161] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4893.666502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4893.667984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4893.671368] Lustre: Skipped 2 previous similar messages [ 4893.675120] Lustre: Skipped 2 previous similar messages [ 4898.780308] Lustre: server umount lustre-MDT0000 complete [ 4898.786195] LustreError: 179844:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4898.791213] LustreError: 179844:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4900.008131] LustreError: 179830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787411670 with bad export cookie 16951430870878597044 [ 4900.011077] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4900.012252] LustreError: 179830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4900.126402] Lustre: server umount lustre-MDT0001 complete [ 4911.637100] Lustre: server umount lustre-OST0000 complete [ 4914.840886] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 4918.217126] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 4929.921143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4930.004956] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4930.015992] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4930.050685] Lustre: lustre-MDT0000: new disk, initializing [ 4930.070081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4930.076140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4931.388762] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4935.372706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4935.410573] Lustre: 185177:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4935.435659] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4935.437172] Lustre: Skipped 1 previous similar message [ 4935.474937] Lustre: lustre-MDT0001: new disk, initializing [ 4935.503175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4935.513947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4935.517565] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4936.764514] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4940.397327] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4942.138217] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4944.356992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4944.468343] Lustre: lustre-OST0000: new disk, initializing [ 4944.470859] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4944.474877] Lustre: 187143:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4944.502688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4946.294753] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4947.764351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4947.766925] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4947.793757] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4949.836435] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4951.226942] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4951.303320] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4952.592392] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4952.662778] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4958.178022] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4958.178604] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4958.180979] LustreError: Skipped 1 previous similar message [ 4958.185306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4958.188112] Lustre: Skipped 1 previous similar message [ 4959.371924] Lustre: server umount lustre-OST0000 complete [ 4961.247846] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4961.250997] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4961.254612] Lustre: Skipped 1 previous similar message [ 4961.256406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4963.296360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4963.296622] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4966.748738] Lustre: server umount lustre-MDT0000 complete [ 4968.040496] LustreError: 185972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787411738 with bad export cookie 16951430870878597744 [ 4968.044739] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4968.045113] LustreError: 185972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4968.164830] Lustre: server umount lustre-MDT0001 complete [ 4970.923334] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 11:15:40 (1787411740) [ 4976.230133] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 4977.178065] Key type lgssc unregistered [ 4977.300393] LNet: 190089:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4977.302313] LNetError: 190089:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4977.311359] LNet: Removed LNI 192.168.206.157@tcp [ 4977.613954] Key type .llcrypt unregistered [ 4977.615037] Key type ._llcrypt unregistered [ 4984.804130] Key type ._llcrypt registered [ 4984.805120] Key type .llcrypt registered [ 4984.852822] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 4985.282918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4985.355617] alg: No test for adler32 (adler32-zlib) [ 4986.205656] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 4986.293349] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 4987.879158] Key type lgssc registered [ 4988.231891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4990.666166] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4990.672644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4991.814890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4993.118044] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4995.851117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4995.978900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4997.275631] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4999.474720] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5001.162698] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5003.548841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5003.618829] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5003.619966] LustreError: 193144:0:(llog.c:1696:llog_backup()) MGC192.168.206.157@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5003.622029] Lustre: 193144:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.206.157@tcp: can't backup local config lustre-OST0000: rc = -28 [ 5003.659048] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5005.491569] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5007.710096] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5010.914273] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5010.917282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5010.919181] Lustre: Skipped 2 previous similar messages [ 5010.921620] Lustre: Skipped 1 previous similar message [ 5014.488684] Lustre: server umount lustre-MDT0000 complete [ 5015.695471] LustreError: 191307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787411786 with bad export cookie 12545408319731612486 [ 5015.698755] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5015.699610] LustreError: 191307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5015.805801] Lustre: server umount lustre-MDT0001 complete [ 5026.268250] Lustre: server umount lustre-OST0000 complete [ 5029.338513] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 5032.451878] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 5043.869161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5043.954321] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5043.963152] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5044.001697] Lustre: lustre-MDT0000: new disk, initializing [ 5044.019883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5044.025565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5045.377209] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5049.636169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5049.678494] Lustre: 196660:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5049.694778] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5049.696141] Lustre: Skipped 1 previous similar message [ 5049.697352] LustreError: 197442:0:(llog.c:1696:llog_backup()) MGC192.168.206.157@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 5049.699772] LustreError: 197442:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 5049.701904] Lustre: 197442:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.157@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 5049.710620] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5049.712421] Lustre: Skipped 1 previous similar message [ 5049.753269] Lustre: lustre-MDT0001: new disk, initializing [ 5049.788561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5049.800695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5049.805182] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5051.156141] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5054.798966] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5056.572244] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5058.780351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5058.884383] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5058.885854] Lustre: Skipped 1 previous similar message [ 5058.887810] LustreError: 198606:0:(llog.c:1696:llog_backup()) MGC192.168.206.157@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5058.891769] LustreError: 198606:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 5058.894416] Lustre: 198606:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.157@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 5058.898112] Lustre: 198606:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 5058.909104] Lustre: lustre-OST0000: new disk, initializing [ 5058.910734] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5058.914477] Lustre: 198629:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5058.948563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5060.879146] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5064.187697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5064.191217] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5064.221742] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5064.611459] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5066.042344] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5066.120188] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5067.679576] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5067.763946] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5069.281737] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5069.281944] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5069.285487] LustreError: Skipped 1 previous similar message [ 5069.293526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5069.296734] Lustre: Skipped 1 previous similar message [ 5070.549414] Lustre: server umount lustre-OST0000 complete [ 5074.401390] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5074.402181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5074.406941] Lustre: Skipped 2 previous similar messages [ 5079.520061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5079.521969] Lustre: Skipped 2 previous similar messages [ 5084.639921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5084.641978] Lustre: Skipped 1 previous similar message [ 5086.175105] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5086.230507] Lustre: server umount lustre-MDT0000 complete [ 5087.451195] LustreError: 196655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787411858 with bad export cookie 12545408319731613193 [ 5087.453422] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5087.456562] LustreError: 196655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5087.559381] Lustre: server umount lustre-MDT0001 complete [ 5090.234532] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 11:17:39 (1787411859) [ 5095.358863] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 5098.860834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5098.991231] LustreError: 201753:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5098.996343] LustreError: 201753:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5099.003163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5100.337839] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5103.113957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5104.616410] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5105.579705] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5107.706480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5109.643418] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5112.423315] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5112.480226] Lustre: lustre-OST0001: new disk, initializing [ 5112.481677] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5112.483870] Lustre: 204059:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5112.898877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5112.901528] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5112.924736] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5114.347653] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5119.688639] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5126.289860] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5126.841492] Lustre: DEBUG MARKER: ADD OST3 [ 5128.383533] loop: module loaded [ 5128.401368] loop0: detected capacity change from 0 to 204800000 [ 5128.470502] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5128.474945] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5128.478896] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5130.474887] loop0: detected capacity change from 0 to 204800000 [ 5132.231989] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5132.292477] Lustre: lustre-OST0003: new disk, initializing [ 5132.293735] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 5132.295859] Lustre: 205790:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5132.328202] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 5132.331105] Lustre: Skipped 3 previous similar messages [ 5133.436915] Lustre: DEBUG MARKER: STOP OST3 [ 5133.971798] Lustre: *** cfs_fail_loc=2109, val=2*** [ 5134.010971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 5134.014689] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 5134.025223] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 5134.137052] Lustre: Failing over lustre-OST0003 [ 5134.173771] Lustre: server umount lustre-OST0003 complete [ 5134.193218] LustreError: 204150:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.206.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5134.433161] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5136.032073] LustreError: 206170:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 5138.363572] Lustre: DEBUG MARKER: START OST3 again [ 5139.429986] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5139.431091] LustreError: 204144:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5139.433734] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5139.433938] Lustre: Skipped 1 previous similar message [ 5139.443088] LustreError: 204144:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5140.574231] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5140.660402] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 5142.559676] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5142.626231] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5142.636683] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5142.636763] Lustre: lustre-OST0003: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5155.805313] Lustre: Failing over lustre-OST0003 [ 5155.840312] Lustre: server umount lustre-OST0003 complete [ 5156.325777] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5156.327500] LustreError: 203240:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5156.329656] Lustre: Skipped 1 previous similar message [ 5156.334193] LustreError: 203240:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5161.444737] LustreError: 204144:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5161.450283] LustreError: 204144:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5170.144045] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5170.146578] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5170.152589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5171.688345] LustreError: 203239:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5171.696550] LustreError: 203239:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5174.502388] Lustre: server umount lustre-MDT0000 complete [ 5175.789151] LustreError: 201736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787411946 with bad export cookie 12545408319731614061 [ 5175.792966] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5175.793302] LustreError: 201736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5182.062734] Lustre: server umount lustre-MDT0001 complete [ 5183.324435] Lustre: server umount lustre-OST0000 complete [ 5184.548059] Lustre: server umount lustre-OST0001 complete [ 5186.470585] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 5188.945988] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 5200.474074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5200.552473] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5200.562910] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5200.596919] Lustre: lustre-MDT0000: new disk, initializing [ 5200.616082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5200.618063] Lustre: Skipped 1 previous similar message [ 5200.622918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5201.894357] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5205.774879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5205.814543] Lustre: 211203:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5205.923424] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5207.180220] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5210.698982] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5212.365705] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5214.499555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5214.596198] Lustre: 213169:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5216.187910] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5216.229263] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5216.520010] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5220.013455] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5221.367399] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5221.439173] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5222.684863] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5222.755832] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5226.465547] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5226.467151] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5226.468296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5226.470837] Lustre: Skipped 6 previous similar messages [ 5226.474095] Lustre: Skipped 9 previous similar messages [ 5229.456524] Lustre: server umount lustre-OST0000 complete [ 5231.585730] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5231.587748] LustreError: Skipped 1 previous similar message [ 5236.761346] Lustre: server umount lustre-MDT0000 complete [ 5237.893289] LustreError: 211194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787412008 with bad export cookie 12545408319731616308 [ 5237.896093] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5237.896844] LustreError: 211194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5238.002976] Lustre: server umount lustre-MDT0001 complete [ 5239.095748] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 11:20:08 (1787412008) [ 5244.082184] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 5246.999765] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 5258.057921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5258.135526] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5258.145056] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5258.147203] Lustre: Skipped 3 previous similar messages [ 5258.179692] Lustre: lustre-MDT0000: new disk, initializing [ 5258.181224] Lustre: Skipped 2 previous similar messages [ 5258.202975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5258.205562] Lustre: Skipped 2 previous similar messages [ 5259.494991] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5263.650547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5263.698488] Lustre: 217771:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5263.817921] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5265.311890] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5269.245346] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5271.082747] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5273.385263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5273.503746] Lustre: 219737:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5275.509238] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5275.564454] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5279.317343] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5280.885356] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5280.971120] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5282.443145] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5282.521467] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5285.265758] Lustre: server umount lustre-OST0000 complete [ 5289.440127] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5289.443866] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5289.448494] Lustre: Skipped 2 previous similar messages [ 5289.450290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5289.452743] Lustre: Skipped 4 previous similar messages [ 5297.152910] Lustre: server umount lustre-MDT0000 complete [ 5298.536630] LustreError: 218565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787412069 with bad export cookie 12545408319731617190 [ 5298.541138] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5298.542080] LustreError: 218565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5308.954967] Lustre: server umount lustre-MDT0001 complete [ 5311.876601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5312.026471] LustreError: 221954:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5312.032428] LustreError: 221954:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5313.307591] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5316.037205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5317.442185] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5319.661664] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5321.373658] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5323.693695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5325.558336] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5327.760640] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5346.271845] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5350.536079] Lustre: server umount lustre-OST0000 complete [ 5357.848472] Lustre: server umount lustre-MDT0000 complete [ 5358.560372] LustreError: 222682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5358.567041] LustreError: 222682:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5359.236438] LustreError: 221934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787412130 with bad export cookie 12545408319731618072 [ 5359.239647] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5359.241889] LustreError: 221934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5359.369768] Lustre: server umount lustre-MDT0001 complete [ 5462.674499] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 5463.759596] Key type lgssc unregistered [ 5463.902445] LNet: 225635:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5463.905198] LNetError: 225635:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5463.911517] LNet: Removed LNI 192.168.206.157@tcp [ 5464.258154] Key type .llcrypt unregistered [ 5464.259944] Key type ._llcrypt unregistered [ 5472.939581] Key type ._llcrypt registered [ 5472.940966] Key type .llcrypt registered [ 5472.981623] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 5479.439253] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 5479.821139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5479.828749] alg: No test for adler32 (adler32-zlib) [ 5480.691671] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 5480.778322] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 5482.359157] Key type lgssc registered [ 5482.732280] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5494.531065] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5494.538762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5495.635524] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5495.646529] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5495.684515] Lustre: lustre-MDT0000: new disk, initializing [ 5495.708398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5495.714542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5497.212971] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5501.419215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5501.468549] Lustre: 228975:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5501.495817] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5501.497588] Lustre: Skipped 1 previous similar message [ 5501.534564] Lustre: lustre-MDT0001: new disk, initializing [ 5501.571834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5501.584761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5501.588575] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5502.975156] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5506.665928] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5508.498475] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5510.783879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5510.896733] Lustre: lustre-OST0000: new disk, initializing [ 5510.899443] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5510.902263] Lustre: 230940:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5510.941437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5512.904137] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5516.619339] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5517.306525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5517.309409] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5517.346245] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5518.164955] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5518.240973] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5519.702642] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5519.782557] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5522.404851] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5522.405548] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5522.413227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5522.415350] Lustre: Skipped 1 previous similar message [ 5526.547682] Lustre: server umount lustre-OST0000 complete [ 5532.641278] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5532.641568] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5532.642151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5532.644864] LustreError: Skipped 1 previous similar message [ 5532.648633] Lustre: Skipped 2 previous similar messages [ 5537.760034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5537.761882] Lustre: Skipped 2 previous similar messages [ 5542.367119] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5542.431780] Lustre: server umount lustre-MDT0000 complete [ 5543.706434] LustreError: 228968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787412314 with bad export cookie 15968617328842985342 [ 5543.710442] LustreError: 228968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5543.711726] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5543.839313] Lustre: server umount lustre-MDT0001 complete [ 5550.158911] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 5553.921058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5554.068186] LustreError: 233869:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5554.074033] LustreError: 233869:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5554.080450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5555.476553] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5558.459913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5558.601624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5560.075858] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5561.070604] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5563.339483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5565.464722] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5568.382655] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5568.449465] Lustre: lustre-OST0001: new disk, initializing [ 5568.451311] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5568.453761] Lustre: 236175:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5568.485312] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5568.487691] Lustre: Skipped 1 previous similar message [ 5570.472542] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5571.084919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5571.090558] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5571.134242] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5576.067676] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5582.820883] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5586.538480] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 11:25:56 (1787412356) [ 5589.471943] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5589.474421] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5589.479096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5591.523896] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5591.527391] Lustre: Skipped 1 previous similar message [ 5593.376875] Lustre: server umount lustre-MDT0000 complete [ 5594.613280] LustreError: 233851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787412365 with bad export cookie 15968617328842986210 [ 5594.616901] LustreError: 233851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5594.616933] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5594.730044] Lustre: server umount lustre-MDT0001 complete [ 5605.831668] Lustre: server umount lustre-OST0000 complete [ 5616.285960] Lustre: server umount lustre-OST0001 complete [ 5618.381180] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 5621.830195] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 5633.664392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5633.756547] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5633.766178] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5633.804661] Lustre: lustre-MDT0000: new disk, initializing [ 5633.827691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5633.835163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5635.287623] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5639.554091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5639.603639] Lustre: 240112:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5639.733183] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5641.205996] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5644.995407] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5646.898380] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5649.174892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5649.289314] Lustre: lustre-OST0000: new disk, initializing [ 5649.290712] Lustre: Skipped 1 previous similar message [ 5649.292255] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5649.294374] Lustre: Skipped 2 previous similar messages [ 5649.296312] Lustre: 242078:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5651.073091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5651.077859] Lustre: Skipped 1 previous similar message [ 5651.081648] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5651.116338] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5651.332298] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5655.086464] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5656.570529] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5656.646313] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5658.080501] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5658.166479] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5661.667722] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5661.668887] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5661.670376] LustreError: Skipped 1 previous similar message [ 5661.670903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5661.674034] Lustre: Skipped 1 previous similar message [ 5661.679936] Lustre: Skipped 5 previous similar messages [ 5664.917062] Lustre: server umount lustre-OST0000 complete [ 5671.905355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5671.909322] Lustre: Skipped 3 previous similar messages [ 5672.423345] Lustre: server umount lustre-MDT0000 complete [ 5673.717635] LustreError: 240104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787412444 with bad export cookie 15968617328842988303 [ 5673.722711] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5673.851193] Lustre: server umount lustre-MDT0001 complete [ 5676.836313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5677.013118] LustreError: 244292:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5677.041286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5677.043360] Lustre: Skipped 2 previous similar messages [ 5678.410118] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5681.261448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5682.724567] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5685.078359] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5686.902645] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5689.229500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5691.414145] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5693.936974] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5839.843174] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5839.844032] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5839.848518] Lustre: Skipped 2 previous similar messages [ 5839.850472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5839.852483] Lustre: Skipped 1 previous similar message [ 5851.103150] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5851.157618] Lustre: server umount lustre-OST0000 complete [ 5858.516225] Lustre: server umount lustre-MDT0000 complete [ 5859.731236] LustreError: 244273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787412630 with bad export cookie 15968617328842989185 [ 5859.734562] LustreError: 244273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5859.735026] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5859.846640] Lustre: server umount lustre-MDT0001 complete [ 5957.828718] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 5958.848137] Key type lgssc unregistered [ 5958.979488] LNet: 247897:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5958.982219] LNetError: 247897:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5958.990509] LNet: Removed LNI 192.168.206.157@tcp [ 5959.346111] Key type .llcrypt unregistered [ 5959.347300] Key type ._llcrypt unregistered [ 5967.492396] Key type ._llcrypt registered [ 5967.493541] Key type .llcrypt registered [ 5967.529231] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 5973.670507] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 5974.063083] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5974.116676] alg: No test for adler32 (adler32-zlib) [ 5974.984672] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 5975.086702] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 5976.671193] Key type lgssc registered [ 5977.087591] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5992.150971] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 5995.635918] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5995.642952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5996.735582] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5996.747117] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5996.789342] Lustre: lustre-MDT0000: new disk, initializing [ 5996.810075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5996.816239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5998.239153] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6002.511465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6002.558893] Lustre: 251937:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6002.584967] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6002.588258] Lustre: Skipped 1 previous similar message [ 6002.637137] Lustre: lustre-MDT0001: new disk, initializing [ 6002.673435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6002.687538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6002.692420] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6004.167756] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6006.503874] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6008.795053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6008.914628] Lustre: lustre-OST0000: new disk, initializing [ 6008.916518] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6008.919410] Lustre: 253569:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6008.954920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6010.978291] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6011.388074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6011.391927] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6011.434543] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6015.078900] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6015.142728] Lustre: lustre-OST0001: new disk, initializing [ 6015.144712] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6015.147597] Lustre: 254440:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6015.181379] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6017.027352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6017.031037] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6017.064478] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6017.133812] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6022.647442] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6027.333722] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6032.356976] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6032.358211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6032.360589] Lustre: Skipped 1 previous similar message [ 6032.364225] Lustre: Skipped 3 previous similar messages [ 6036.384929] Lustre: server umount lustre-MDT0000 complete [ 6037.615833] LustreError: 251930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787412808 with bad export cookie 3836986624199534693 [ 6037.619065] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6037.620427] LustreError: 251930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6037.735905] Lustre: server umount lustre-MDT0001 complete [ 6048.775173] Lustre: server umount lustre-OST0000 complete [ 6059.300030] Lustre: server umount lustre-OST0001 complete [ 6063.971421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6067.107181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6069.945492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6073.235239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6079.759515] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 6083.396573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6083.403133] Lustre: lustre-MDT0000: reset Object Index mappings [ 6083.556900] LustreError: 259439:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6083.562713] LustreError: 259439:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6083.574919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6084.988496] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6087.867773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6087.874407] Lustre: lustre-MDT0001: reset Object Index mappings [ 6089.496453] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6090.470315] Lustre: 260581:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6092.683077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6092.834266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6092.836311] Lustre: Skipped 1 previous similar message [ 6093.859818] LustreError: 260936:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6093.864161] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6094.908737] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6097.719281] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6098.862107] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 6099.748604] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6102.896609] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6104.098744] Lustre: 262447:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6108.639941] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6108.643425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6108.647021] Lustre: Skipped 2 previous similar messages [ 6108.648876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6111.384201] Lustre: server umount lustre-MDT0000 complete [ 6112.604473] LustreError: 259421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787412883 with bad export cookie 3836986624199541469 [ 6112.607364] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6112.609301] LustreError: 259421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6112.736109] Lustre: server umount lustre-MDT0001 complete [ 6123.464909] Lustre: server umount lustre-OST0000 complete [ 6134.951461] Lustre: server umount lustre-OST0001 complete [ 6141.295389] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 11:35:10 (1787412910) [ 6146.629067] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 6150.019226] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 6162.016194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6162.104496] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6162.115115] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6162.149988] Lustre: lustre-MDT0000: new disk, initializing [ 6162.173199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6162.176135] Lustre: Skipped 1 previous similar message [ 6162.183604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6163.598383] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6167.728468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6167.778903] Lustre: 266696:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6167.879791] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6169.213833] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6172.925934] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6174.757950] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6177.020278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6177.144346] Lustre: lustre-OST0000: new disk, initializing [ 6177.145777] Lustre: Skipped 1 previous similar message [ 6177.147228] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6177.149093] Lustre: Skipped 2 previous similar messages [ 6177.151712] Lustre: 268664:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6178.878781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6178.882220] Lustre: Skipped 1 previous similar message [ 6178.884543] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6178.927499] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6179.236862] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6182.922414] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6184.327081] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6184.404367] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6185.822184] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6185.889195] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6189.023885] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6189.026542] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6189.031323] Lustre: Skipped 3 previous similar messages [ 6189.034447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6189.037481] Lustre: Skipped 4 previous similar messages [ 6192.595465] Lustre: server umount lustre-OST0000 complete [ 6194.656619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6194.657320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6194.660310] Lustre: Skipped 2 previous similar messages [ 6194.662101] Lustre: Skipped 1 previous similar message [ 6199.776789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6199.778941] Lustre: Skipped 2 previous similar messages [ 6199.912161] Lustre: server umount lustre-MDT0000 complete [ 6201.127108] LustreError: 266690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787412971 with bad export cookie 3836986624199543541 [ 6201.130330] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6201.130626] LustreError: 266690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6201.244896] Lustre: server umount lustre-MDT0001 complete [ 6207.444343] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 6211.239536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6211.395251] LustreError: 271589:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6211.409304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6211.412970] Lustre: Skipped 2 previous similar messages [ 6212.776235] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6215.726826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6217.288788] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6218.324362] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6220.540826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6222.496438] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6225.298047] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6225.359552] Lustre: lustre-OST0001: new disk, initializing [ 6225.361724] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6225.364959] Lustre: 273896:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6227.213249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6227.215941] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6227.259287] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6227.361577] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6231.713997] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6232.919439] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6234.565741] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 6234.570764] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 6236.640202] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6236.642700] LustreError: Skipped 1 previous similar message [ 6236.644614] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6236.648204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6242.410085] Lustre: server umount lustre-MDT0000 complete [ 6242.788610] LustreError: 273261:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6242.793499] LustreError: 273261:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6243.620370] LustreError: 272326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787413014 with bad export cookie 3836986624199544423 [ 6243.623577] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6243.625156] LustreError: 272326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6243.727322] Lustre: server umount lustre-MDT0001 complete [ 6255.111066] Lustre: server umount lustre-OST0000 complete [ 6266.591827] Lustre: server umount lustre-OST0001 complete [ 6271.912685] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 6275.385213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6275.516590] LustreError: 276948:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6275.534457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6275.537633] Lustre: Skipped 3 previous similar messages [ 6276.837400] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6279.456320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6280.888736] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6281.771156] Lustre: 278088:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6283.846023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6285.762108] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6286.052945] LustreError: 278444:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6286.056913] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 6288.525664] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6289.662591] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 6290.556758] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6293.673607] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6294.939047] Lustre: 279962:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6296.664506] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 6296.668903] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 6299.586050] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 11:37:49 (1787413069) [ 6315.488431] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6315.489993] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6315.492849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6315.494090] Lustre: Skipped 6 previous similar messages [ 6315.495713] Lustre: Skipped 6 previous similar messages [ 6317.208403] Lustre: server umount lustre-MDT0000 complete [ 6318.355687] LustreError: 276929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787413089 with bad export cookie 3836986624199550744 [ 6318.358036] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6318.359511] LustreError: 276929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6320.607946] LustreError: 276943:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6320.613522] LustreError: 276943:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6324.527153] Lustre: server umount lustre-MDT0001 complete [ 6340.063066] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6340.094677] Lustre: server umount lustre-OST0000 complete [ 6347.352361] Lustre: server umount lustre-OST0001 complete [ 6349.476117] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 6352.629700] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 6366.172925] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 6369.313074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6369.396267] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6369.406398] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6369.439474] Lustre: lustre-MDT0000: new disk, initializing [ 6369.462316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6370.696906] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6374.566580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6374.610213] Lustre: 284190:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6374.713657] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6375.998655] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6380.588021] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6380.727229] Lustre: 285825:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6381.235299] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6382.768738] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6387.030904] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6389.062044] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6389.283035] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6393.916270] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6399.286069] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 11:39:28 (1787413168) [ 6400.480160] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6400.483720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6400.489170] Lustre: Skipped 2 previous similar messages [ 6400.491311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6400.493784] Lustre: Skipped 5 previous similar messages [ 6406.109504] Lustre: server umount lustre-MDT0000 complete [ 6407.259230] LustreError: 284183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787413178 with bad export cookie 3836986624199558626 [ 6407.262735] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6407.264183] LustreError: 284183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6407.373411] Lustre: server umount lustre-MDT0001 complete [ 6418.879496] Lustre: server umount lustre-OST0000 complete [ 6430.242401] Lustre: server umount lustre-OST0001 complete [ 6433.253104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6433.400810] LustreError: 288936:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6433.406183] LustreError: 288936:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6433.427380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6433.429771] Lustre: Skipped 7 previous similar messages [ 6434.701695] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6437.417256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6438.855370] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6441.017726] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6442.710114] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6444.844659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6446.803633] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6449.030836] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6465.359301] loop1: detected capacity change from 0 to 204800000 [ 6465.446251] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6465.452485] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6465.458686] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6467.117688] loop1: detected capacity change from 0 to 204800000 [ 6469.056864] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6469.099721] LustreError: 288926:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs14261: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 6469.105143] LustreError: 288926:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs14261 [ 6469.108805] LustreError: 288926:0:(mgs_handler.c:517:mgs_target_reg()) fs14261-MDT0001: Error checking target index: rc = -13 [ 6469.114574] LustreError: fs14261-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 6469.120195] LustreError: 292677:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 6469.124275] LustreError: 292677:0:(tgt_mount.c:2129:server_put_super()) no obd fs14261-MDT0001 [ 6469.127803] LustreError: 292677:0:(tgt_mount.c:116:server_deregister_mount()) fs14261-MDT0001 not registered [ 6469.141598] Lustre: server umount fs14261-MDT0001 complete [ 6469.144117] LustreError: 292677:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 6471.940219] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6471.969230] Lustre: Setting parameter fs14261-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs14261-MDT0001 [ 6471.973961] Lustre: Skipped 2 previous similar messages [ 6471.991172] Lustre: srv-fs14261-MDT0001: No data found on store. Initialize space. [ 6471.994170] Lustre: Skipped 4 previous similar messages [ 6472.035463] Lustre: fs14261-MDT0001: new disk, initializing [ 6472.036839] Lustre: Skipped 3 previous similar messages [ 6472.067099] Lustre: srv-fs14261-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 6472.069458] Lustre: 293141:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs14261-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6472.072122] LustreError: 293141:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs14261-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6473.375115] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6476.193226] Lustre: server umount fs14261-MDT0001 complete [ 6480.521183] loop1: detected capacity change from 0 to 204800000 [ 6482.518301] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6482.631664] Lustre: 294707:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs14261-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6482.634944] LustreError: 294707:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs14261-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6483.935994] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6485.464687] Lustre: server umount fs14261-MDT0001 complete [ 6489.058417] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6489.058442] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6489.059611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6489.059616] Lustre: Skipped 4 previous similar messages [ 6489.061917] Lustre: Skipped 5 previous similar messages [ 6494.867380] Lustre: server umount lustre-MDT0000 complete [ 6497.531636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6497.569514] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6497.626896] LustreError: 290765:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6497.631465] LustreError: 290765:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 25 previous similar messages [ 6498.908250] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6501.245869] loop1: detected capacity change from 0 to 204800000 [ 6502.886955] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6502.984613] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6503.098015] Lustre: 297149:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs14261-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6503.101519] LustreError: 297149:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs14261-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6504.384663] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6505.997508] Lustre: server umount fs14261-MDT0001 complete [ 6513.120254] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6518.353100] Lustre: server umount lustre-OST0000 complete [ 6525.789446] Lustre: server umount lustre-MDT0000 complete [ 6527.062283] LustreError: 290790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787413297 with bad export cookie 3836986624199561300 [ 6527.065671] LustreError: 290790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6527.180883] Lustre: server umount lustre-MDT0001 complete [ 6530.614849] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 6531.596601] Key type lgssc unregistered [ 6531.724297] LNet: 299490:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6531.727176] LNetError: 299490:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6531.738346] LNet: Removed LNI 192.168.206.157@tcp [ 6532.049098] Key type .llcrypt unregistered [ 6532.050125] Key type ._llcrypt unregistered [ 6539.054399] Key type ._llcrypt registered [ 6539.055341] Key type .llcrypt registered [ 6539.088761] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 6539.488147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6539.495378] alg: No test for adler32 (adler32-zlib) [ 6540.342363] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 6540.424579] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 6541.999148] Key type lgssc registered [ 6542.321748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6544.655159] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6544.660534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6545.790138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6546.993346] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6549.479840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6549.604802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6550.836195] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6553.020707] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6554.804858] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6556.872643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6556.974127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6558.872896] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6560.037755] LustreError: 302563:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6560.044745] LustreError: 302563:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6561.262682] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6565.348664] LustreError: 303096:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6570.466115] LustreError: 303096:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6570.473446] LustreError: 303096:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6575.589981] LustreError: 302563:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6575.597191] LustreError: 302563:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6577.624716] loop1: detected capacity change from 0 to 204800000 [ 6577.689014] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6577.693989] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6577.700258] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6579.405114] loop1: detected capacity change from 0 to 204800000 [ 6580.713425] LustreError: 302565:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6580.718472] LustreError: 302565:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6581.358522] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6581.410318] Lustre: fs241-OST0000: new disk, initializing [ 6581.412324] Lustre: srv-fs241-OST0000: No data found on store. Initialize space. [ 6581.414706] Lustre: 304490:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6581.438888] Lustre: fs241-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6583.469199] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6587.081462] Lustre: server umount fs241-OST0000 complete [ 6590.945448] LustreError: 302563:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6590.953048] LustreError: 302563:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6591.364209] loop1: detected capacity change from 0 to 204800000 [ 6593.319734] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6593.387824] Lustre: fs241-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6595.407709] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6598.274515] loop2: detected capacity change from 0 to 204800000 [ 6598.367137] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6598.370302] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6598.374394] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6600.134583] loop2: detected capacity change from 0 to 204800000 [ 6602.107819] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6602.139960] LustreError: fs241-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 6602.143835] LustreError: fs241-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 6602.147898] LustreError: 307526:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 6602.151344] LustreError: 307526:0:(tgt_mount.c:2129:server_put_super()) no obd fs241-OST0001 [ 6602.153772] LustreError: 307526:0:(tgt_mount.c:116:server_deregister_mount()) fs241-OST0001 not registered [ 6602.163902] Lustre: server umount fs241-OST0001 complete [ 6602.165397] LustreError: 307526:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 6605.513327] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6605.564586] Lustre: fs241-OST0001: new disk, initializing [ 6605.565988] Lustre: srv-fs241-OST0001: No data found on store. Initialize space. [ 6605.568059] Lustre: 308076:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6605.594591] Lustre: fs241-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6607.367262] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6607.455133] LustreError: 303093:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6607.461179] LustreError: 303093:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 6610.796990] Lustre: server umount fs241-OST0000 complete [ 6613.819980] Lustre: server umount fs241-OST0001 complete [ 6617.927689] loop1: detected capacity change from 0 to 204800000 [ 6619.790212] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6621.717428] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6623.816570] loop2: detected capacity change from 0 to 204800000 [ 6625.477479] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6625.527904] Lustre: fs241-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6625.529775] Lustre: Skipped 1 previous similar message [ 6627.390394] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6630.318320] Lustre: server umount fs241-OST0000 complete [ 6633.513888] Lustre: server umount fs241-OST0001 complete [ 6642.145676] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6642.146536] LustreError: 302565:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6642.147389] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6642.148127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6642.149023] LustreError: Skipped 1 previous similar message [ 6642.153990] LustreError: 302565:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [ 6643.280061] Lustre: server umount lustre-OST0000 complete [ 6646.751828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6646.755061] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6646.759749] Lustre: Skipped 1 previous similar message [ 6646.761497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6646.763875] Lustre: Skipped 1 previous similar message [ 6650.645273] Lustre: server umount lustre-MDT0000 complete [ 6651.842665] LustreError: 300718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787413422 with bad export cookie 7045101536447865904 [ 6651.845594] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6651.848009] LustreError: 300718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6651.989688] Lustre: server umount lustre-MDT0001 complete [ 6655.452049] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 6656.476852] Key type lgssc unregistered [ 6656.606435] LNet: 313986:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6656.610259] LNetError: 313986:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6656.616289] LNet: Removed LNI 192.168.206.157@tcp [ 6656.948113] Key type .llcrypt unregistered [ 6656.949244] Key type ._llcrypt unregistered [ 6666.725496] Key type ._llcrypt registered [ 6666.726925] Key type .llcrypt registered [ 6666.762254] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 6672.906656] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 6673.424767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6673.445361] alg: No test for adler32 (adler32-zlib) [ 6674.294238] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 6674.390775] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 6675.975239] Key type lgssc registered [ 6676.351739] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6687.967372] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6687.974017] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6689.049776] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6689.058670] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6689.094382] Lustre: lustre-MDT0000: new disk, initializing [ 6689.118165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6689.123932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6690.480549] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6694.478251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6694.520102] Lustre: 317815:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6694.543558] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6694.546030] Lustre: Skipped 1 previous similar message [ 6694.578184] Lustre: lustre-MDT0001: new disk, initializing [ 6694.604879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6694.614591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6694.617554] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6695.854115] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6699.510589] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6701.339563] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6703.625820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6703.735652] Lustre: lustre-OST0000: new disk, initializing [ 6703.737688] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6703.740613] Lustre: 319784:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6703.772320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6705.734937] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6705.791172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6705.795372] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6705.841653] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6709.441193] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6710.129552] Lustre: Mounted lustre-client [ 6710.454271] Lustre: Unmounted lustre-client [ 6715.364513] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6715.365529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6715.368335] Lustre: Skipped 1 previous similar message [ 6719.443406] Lustre: server umount lustre-OST0000 complete [ 6725.601905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6725.602034] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6725.609213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6725.611082] Lustre: Skipped 2 previous similar messages [ 6730.719904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6730.721529] Lustre: Skipped 1 previous similar message [ 6735.327095] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6735.389492] Lustre: server umount lustre-MDT0000 complete [ 6736.527124] LustreError: 317809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787413507 with bad export cookie 14319891497051521158 [ 6736.530525] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6736.533061] LustreError: 317809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6736.639659] Lustre: server umount lustre-MDT0001 complete [ 6740.028546] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 6740.967944] Key type lgssc unregistered [ 6766.687296] LNet: 321943:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6766.690046] LNetError: 321943:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6841.445355] LNet: Removed LNI 192.168.206.157@tcp [ 6841.777126] Key type .llcrypt unregistered [ 6841.778336] Key type ._llcrypt unregistered [ 6847.628720] Key type ._llcrypt registered [ 6847.630011] Key type .llcrypt registered [ 6847.672991] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 6853.640641] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 6853.996675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6854.024191] alg: No test for adler32 (adler32-zlib) [ 6854.862245] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 6854.945881] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 6856.519168] Key type lgssc registered [ 6856.882897] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6865.935633] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 11:47:15 (1787413635) [ 6867.983347] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6867.988062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6869.060740] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6869.069971] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6869.102503] Lustre: lustre-MDT0000: new disk, initializing [ 6869.120398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6869.125691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6870.316151] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6874.196705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6874.239434] Lustre: 325141:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6874.261672] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6874.263452] Lustre: Skipped 1 previous similar message [ 6874.296455] Lustre: lustre-MDT0001: new disk, initializing [ 6874.327305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6874.339957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6874.343282] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6875.560870] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6879.240376] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6880.881879] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6882.964041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6883.054527] Lustre: lustre-OST0000: new disk, initializing [ 6883.056448] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6883.059053] Lustre: 327107:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6883.085035] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6884.663058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6884.666934] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6884.701522] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6884.895236] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6888.570897] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6895.073654] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6895.075216] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6895.075986] LustreError: Skipped 1 previous similar message [ 6895.079905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6895.082558] Lustre: Skipped 1 previous similar message [ 6900.110296] Lustre: server umount lustre-OST0000 complete [ 6905.312676] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6905.312751] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6905.313480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6905.323108] Lustre: Skipped 2 previous similar messages [ 6907.416552] Lustre: server umount lustre-MDT0000 complete [ 6908.606384] LustreError: 325132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787413679 with bad export cookie 5703647917755710993 [ 6908.610433] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6908.610556] LustreError: 325132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6908.730323] Lustre: server umount lustre-MDT0001 complete [ 6912.083391] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 6913.005964] Key type lgssc unregistered [ 6913.119338] LNet: 329012:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6913.121400] LNetError: 329012:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6913.127287] LNet: Removed LNI 192.168.206.157@tcp [ 6913.417601] Key type .llcrypt unregistered [ 6913.418563] Key type ._llcrypt unregistered [ 6920.780295] Key type ._llcrypt registered [ 6920.781245] Key type .llcrypt registered [ 6920.811441] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 6921.058067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6921.068000] alg: No test for adler32 (adler32-zlib) [ 6921.919074] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 6922.003362] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 6923.575126] Key type lgssc registered [ 6923.900021] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6926.229428] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6926.236607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6927.367773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6928.667256] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6931.251491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6931.368680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6932.651068] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6934.850751] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6936.412382] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6938.440760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6938.538421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6940.315770] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6942.456726] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6956.306217] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6956.827589] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6958.048603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6958.050319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6958.052189] Lustre: Skipped 2 previous similar messages [ 6958.053650] Lustre: Skipped 1 previous similar message [ 6960.147573] Lustre: server umount lustre-MDT0000 complete [ 6961.297203] LustreError: 332105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787413732 with bad export cookie 12122505210913136710 [ 6961.300249] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6961.300658] LustreError: 332105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6963.167886] LustreError: 330991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6963.168117] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6963.173130] LustreError: 330991:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6963.179120] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6963.180640] Lustre: Skipped 2 previous similar messages [ 6966.175994] LustreError: 330991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6966.176050] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6966.182582] LustreError: 330991:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6966.185514] Lustre: Skipped 1 previous similar message [ 6967.467949] Lustre: server umount lustre-MDT0001 complete [ 6973.540156] Lustre: lustre-OST0000: Not available for connect from 192.168.206.57@tcp (stopping) [ 6974.743143] Lustre: server umount lustre-OST0000 complete [ 6977.425602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6978.700799] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6979.549279] Lustre: 334458:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.157@tcp [ 6980.191604] Lustre: server umount MGS complete [ 6982.775293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6982.904901] LustreError: 335030:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6982.928530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6984.251119] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6986.950031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6987.064093] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6988.248645] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6990.916423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6992.864361] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6995.118212] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6995.615046] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6997.255487] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6997.762215] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7002.594111] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7002.594502] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7002.598317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7002.601833] Lustre: Skipped 1 previous similar message [ 7005.709454] Lustre: server umount lustre-OST0000 complete [ 7007.712662] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7007.713042] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7007.718524] Lustre: Skipped 2 previous similar messages [ 7012.832407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7012.833946] Lustre: Skipped 3 previous similar messages [ 7013.074552] Lustre: server umount lustre-MDT0000 complete [ 7014.171156] LustreError: 335011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787413785 with bad export cookie 12122505210913137536 [ 7014.173980] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7014.174435] LustreError: 335011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7014.277662] Lustre: server umount lustre-MDT0001 complete [ 7017.558168] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 7018.484873] Key type lgssc unregistered [ 7018.604381] LNet: 338461:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7018.606646] LNetError: 338461:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7018.614250] LNet: Removed LNI 192.168.206.157@tcp [ 7018.882754] Key type .llcrypt unregistered [ 7018.883736] Key type ._llcrypt unregistered [ 7024.315980] Key type ._llcrypt registered [ 7024.317057] Key type .llcrypt registered [ 7024.355983] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 7031.135305] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 7031.441775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7031.485174] alg: No test for adler32 (adler32-zlib) [ 7032.345776] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 7032.433129] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 7034.007073] Key type lgssc registered [ 7034.374292] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7045.178701] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7045.184892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7046.261091] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7046.271670] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7046.314739] Lustre: lustre-MDT0000: new disk, initializing [ 7046.335511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7046.341279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7047.650445] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7051.710234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7051.761493] Lustre: 341594:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7051.786308] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7051.789461] Lustre: Skipped 1 previous similar message [ 7051.832383] Lustre: lustre-MDT0001: new disk, initializing [ 7051.866711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7051.879848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7051.883198] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7053.226016] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7056.797070] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7058.567911] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7060.827512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7060.934692] Lustre: lustre-OST0000: new disk, initializing [ 7060.936872] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7060.940210] Lustre: 343563:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7060.970356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7062.921164] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7066.106892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7066.112126] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7066.140484] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7066.656730] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7068.124789] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7068.198564] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7069.652615] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7069.725993] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7071.202444] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7071.205054] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7071.206421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7071.209645] Lustre: Skipped 1 previous similar message [ 7072.470439] Lustre: server umount lustre-OST0000 complete [ 7076.321798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7076.322408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7076.327859] Lustre: Skipped 1 previous similar message [ 7076.329687] Lustre: Skipped 1 previous similar message [ 7081.440381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7081.442874] Lustre: Skipped 2 previous similar messages [ 7086.560191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7086.562930] Lustre: Skipped 1 previous similar message [ 7088.095066] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7088.164855] Lustre: server umount lustre-MDT0000 complete [ 7089.437616] LustreError: 341584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787413860 with bad export cookie 12114981822644193690 [ 7089.441149] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7089.441681] LustreError: 341584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7089.567685] Lustre: server umount lustre-MDT0001 complete [ 7092.485718] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 11:51:02 (1787413862) [ 7097.950760] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 7101.165029] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 7112.658462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7112.730245] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7112.740066] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7112.772507] Lustre: lustre-MDT0000: new disk, initializing [ 7112.790939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7112.797851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7113.998135] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7117.878842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7117.920562] Lustre: 348312:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7117.941940] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7117.944218] Lustre: Skipped 1 previous similar message [ 7117.979096] Lustre: lustre-MDT0001: new disk, initializing [ 7118.010420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7118.021198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7118.024520] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7119.241531] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7122.728621] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7124.323219] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7126.323060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7126.408402] Lustre: lustre-OST0000: new disk, initializing [ 7126.409907] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7126.412157] Lustre: 350280:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7126.438344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7128.065944] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7131.629164] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7132.669383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7132.672590] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7132.713978] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7148.001337] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7148.001900] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7148.004665] LustreError: Skipped 2 previous similar messages [ 7148.010380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7148.013471] Lustre: Skipped 1 previous similar message [ 7152.084210] Lustre: server umount lustre-OST0000 complete [ 7154.143796] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7154.147356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7154.152957] Lustre: Skipped 1 previous similar message [ 7156.192373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7156.195850] Lustre: Skipped 4 previous similar messages [ 7159.459175] Lustre: server umount lustre-MDT0000 complete [ 7160.683635] LustreError: 350279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787413931 with bad export cookie 12114981822644194544 [ 7160.688120] LustreError: 350279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7160.689039] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7160.812935] Lustre: server umount lustre-MDT0001 complete [ 7164.423823] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 7165.413501] Key type lgssc unregistered [ 7165.534390] LNet: 352430:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7165.538173] LNetError: 352430:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7165.547346] LNet: Removed LNI 192.168.206.157@tcp [ 7165.836545] Key type .llcrypt unregistered [ 7165.837649] Key type ._llcrypt unregistered [ 7195.653287] Key type ._llcrypt registered [ 7195.654572] Key type .llcrypt registered [ 7195.691739] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 7195.783443] Key type .llcrypt unregistered [ 7195.784570] Key type ._llcrypt unregistered [ 7221.414700] Key type ._llcrypt registered [ 7221.415722] Key type .llcrypt registered [ 7221.451654] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 7221.542403] Key type .llcrypt unregistered [ 7221.543587] Key type ._llcrypt unregistered [ 7231.820570] Key type ._llcrypt registered [ 7231.821587] Key type .llcrypt registered [ 7231.853098] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 7231.942444] Key type .llcrypt unregistered [ 7231.943844] Key type ._llcrypt unregistered [ 7242.147741] Key type ._llcrypt registered [ 7242.149178] Key type .llcrypt registered [ 7242.183650] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 7242.280811] Key type .llcrypt unregistered [ 7242.281958] Key type ._llcrypt unregistered [ 7268.162959] Key type ._llcrypt registered [ 7268.163978] Key type .llcrypt registered [ 7268.196803] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 7268.284322] Key type .llcrypt unregistered [ 7268.285538] Key type ._llcrypt unregistered [ 7278.803555] Key type ._llcrypt registered [ 7278.804809] Key type .llcrypt registered [ 7278.835329] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 7278.927232] Key type .llcrypt unregistered [ 7278.928668] Key type ._llcrypt unregistered [ 7289.122181] Key type ._llcrypt registered [ 7289.123069] Key type .llcrypt registered [ 7289.160991] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 7289.267450] Key type .llcrypt unregistered [ 7289.268636] Key type ._llcrypt unregistered [ 7299.634220] Key type ._llcrypt registered [ 7299.635125] Key type .llcrypt registered [ 7299.666721] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 7299.757426] Key type .llcrypt unregistered [ 7299.759148] Key type ._llcrypt unregistered [ 7309.794513] Key type ._llcrypt registered [ 7309.795472] Key type .llcrypt registered [ 7309.827878] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 7309.914175] Key type .llcrypt unregistered [ 7309.915552] Key type ._llcrypt unregistered [ 7320.692851] Key type ._llcrypt registered [ 7320.693937] Key type .llcrypt registered [ 7320.730295] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 7320.832195] Key type .llcrypt unregistered [ 7320.833324] Key type ._llcrypt unregistered [ 7342.154677] Key type ._llcrypt registered [ 7342.155766] Key type .llcrypt registered [ 7342.198679] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 7342.299409] Key type .llcrypt unregistered [ 7342.301201] Key type ._llcrypt unregistered [ 7353.448604] Key type ._llcrypt registered [ 7353.449921] Key type .llcrypt registered [ 7353.488684] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 7353.587173] Key type .llcrypt unregistered [ 7353.588843] Key type ._llcrypt unregistered [ 7364.743330] Key type ._llcrypt registered [ 7364.745145] Key type .llcrypt registered [ 7364.782843] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 7364.879322] Key type .llcrypt unregistered [ 7364.880529] Key type ._llcrypt unregistered [ 7383.029438] Key type ._llcrypt registered [ 7383.031313] Key type .llcrypt registered [ 7383.068863] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 7389.255724] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 7389.588318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7389.631393] alg: No test for adler32 (adler32-zlib) [ 7390.493734] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 7390.582517] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 7392.167179] Key type lgssc registered [ 7392.502845] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7403.746185] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7403.751491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7404.837896] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 7404.846677] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 7404.882149] Lustre: tst250fs-MDT0000: new disk, initializing [ 7404.902093] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7404.908038] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7406.223162] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7410.388773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7410.431148] Lustre: 372333:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7410.463699] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 7410.465864] Lustre: Skipped 1 previous similar message [ 7410.500619] Lustre: tst250fs-MDT0001: new disk, initializing [ 7410.531060] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7410.543982] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7410.548113] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7411.794066] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7413.989224] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7416.215872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7416.345765] Lustre: tst250fs-OST0000: new disk, initializing [ 7416.348609] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 7416.352900] Lustre: 373966:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7416.394529] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7418.360341] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7419.069273] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7419.073409] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7419.127883] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7422.455742] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7422.528069] Lustre: tst250fs-OST0001: new disk, initializing [ 7422.530119] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 7422.533250] Lustre: 374840:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7422.569943] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7424.543240] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 7427.036357] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 7427.084031] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7427.088710] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7427.144238] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7437.283316] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7437.286214] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 7437.287046] Lustre: Skipped 2 previous similar messages [ 7437.288971] Lustre: Skipped 1 previous similar message [ 7442.401168] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 7442.403100] Lustre: Skipped 2 previous similar messages [ 7443.228613] Lustre: server umount tst250fs-MDT0000 complete [ 7444.395317] LustreError: 372372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787414215 with bad export cookie 15597935977236209103 [ 7444.398566] LustreError: 372372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7444.398806] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7444.503765] Lustre: server umount tst250fs-MDT0001 complete [ 7455.755312] Lustre: server umount tst250fs-OST0000 complete [ 7467.231070] Lustre: server umount tst250fs-OST0001 complete [ 7473.137630] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 7475.920473] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 7485.949961] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 11:57:35 (1787414255) [ 7486.488808] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 7491.842755] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 7495.507181] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 7505.942490] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7233 sec ========== 11:57:55 (1787414275) [ 7506.478539] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 11:57:56 (1787414276) === [ 7507.604663] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 11:57:57 (1787414277) === [ 7521.120388] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 7522.170262] Key type lgssc unregistered [ 7522.296296] LNet: 383956:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7522.298739] LNetError: 383956:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7522.307316] LNet: Removed LNI 192.168.206.157@tcp [ 7522.619114] Key type .llcrypt unregistered [ 7522.620337] Key type ._llcrypt unregistered