[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 493164646 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002388] x2apic enabled [ 0.003014] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.006731] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007017] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009008] pid_max: default: 32768 minimum: 301 [ 0.010129] LSM: Security Framework initializing [ 0.011053] Yama: becoming mindful. [ 0.012036] SELinux: Initializing. [ 0.013065] *** VALIDATE selinux *** [ 0.020792] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025286] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026134] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028023] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029106] *** VALIDATE tmpfs *** [ 0.031434] *** VALIDATE proc *** [ 0.033039] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.036118] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037145] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040013] Spectre V2 : User space: Vulnerable [ 0.041010] Speculative Store Bypass: Vulnerable [ 0.044111] debug: unmapping init [mem 0xffffffffb4659000-0xffffffffb4660fff] [ 0.046157] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047668] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048018] ... version: 2 [ 0.049012] ... bit width: 48 [ 0.050011] ... generic registers: 4 [ 0.051009] ... value mask: 0000ffffffffffff [ 0.052014] ... max period: 00007fffffffffff [ 0.053012] ... fixed-purpose events: 3 [ 0.054011] ... event mask: 000000070000000f [ 0.055300] rcu: Hierarchical SRCU implementation. [ 0.057414] smp: Bringing up secondary CPUs ... [ 0.058548] x86: Booting SMP configuration: [ 0.059028] .... node #0, CPUs: #1 #2 #3 [ 0.062750] smp: Brought up 1 node, 4 CPUs [ 0.064013] smpboot: Max logical packages: 1 [ 0.065017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.239698] node 0 deferred pages initialised in 172ms [ 0.243244] devtmpfs: initialized [ 0.244226] x86/mm: Memory block size: 128MB [ 0.246620] gcov: version magic: 0x41383552 [ 0.248043] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.249076] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.250263] pinctrl core: initialized pinctrl subsystem [ 0.251199] [ 0.251869] ************************************************************* [ 0.252011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.253013] ** ** [ 0.254016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.255011] ** ** [ 0.256026] ** This means that this kernel is built to expose internal ** [ 0.257013] ** IOMMU data structures, which may compromise security on ** [ 0.258013] ** your system. ** [ 0.259011] ** ** [ 0.260012] ** If you see this message and you are not debugging the ** [ 0.261014] ** kernel, report this immediately to your vendor! ** [ 0.262012] ** ** [ 0.263015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.264012] ************************************************************* [ 0.265786] NET: Registered protocol family 16 [ 0.266452] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.267063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.268056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.269431] cpuidle: using governor menu [ 0.271850] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.275008] PCI: Using configuration type 1 for base access [ 0.278148] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.288052] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.289033] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.290171] cryptd: max_cpu_qlen set to 1000 [ 0.292076] ACPI: Added _OSI(Module Device) [ 0.293015] ACPI: Added _OSI(Processor Device) [ 0.294022] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.295019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.301322] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.308600] ACPI: Interpreter enabled [ 0.310054] ACPI: PM: (supports S0 S3 S4 S5) [ 0.311009] ACPI: Using IOAPIC for interrupt routing [ 0.313113] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.317397] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.327000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.329040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.332019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.335078] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.341495] acpiphp: Slot [2] registered [ 0.343204] acpiphp: Slot [5] registered [ 0.344184] acpiphp: Slot [6] registered [ 0.346154] acpiphp: Slot [7] registered [ 0.348177] acpiphp: Slot [8] registered [ 0.349162] acpiphp: Slot [9] registered [ 0.351190] acpiphp: Slot [10] registered [ 0.352247] acpiphp: Slot [3] registered [ 0.354124] acpiphp: Slot [4] registered [ 0.356150] acpiphp: Slot [11] registered [ 0.357118] acpiphp: Slot [12] registered [ 0.359124] acpiphp: Slot [13] registered [ 0.360125] acpiphp: Slot [14] registered [ 0.362161] acpiphp: Slot [15] registered [ 0.363131] acpiphp: Slot [16] registered [ 0.365178] acpiphp: Slot [17] registered [ 0.367127] acpiphp: Slot [18] registered [ 0.368124] acpiphp: Slot [19] registered [ 0.370175] acpiphp: Slot [20] registered [ 0.372124] acpiphp: Slot [21] registered [ 0.373182] acpiphp: Slot [22] registered [ 0.375173] acpiphp: Slot [23] registered [ 0.376087] acpiphp: Slot [24] registered [ 0.378152] acpiphp: Slot [25] registered [ 0.380130] acpiphp: Slot [26] registered [ 0.381134] acpiphp: Slot [27] registered [ 0.383150] acpiphp: Slot [28] registered [ 0.384132] acpiphp: Slot [29] registered [ 0.386121] acpiphp: Slot [30] registered [ 0.388146] acpiphp: Slot [31] registered [ 0.389095] PCI host bridge to bus 0000:00 [ 0.391028] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.393037] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.396035] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.399036] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.402034] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.404032] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.406223] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.409088] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.412609] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.422680] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.427715] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.430019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.432015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.434020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.437705] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.440828] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.443053] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.447059] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.453026] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.467025] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.472023] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.479140] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.486017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.496014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.518016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.527000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.537023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.546024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.564026] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.578220] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.584016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.590078] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.609016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.621090] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.626014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.633015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.651014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.661944] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.666014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.672015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.689014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.700268] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.705013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.711014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.731015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.741861] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.746415] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.750394] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.753306] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.755313] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.760034] iommu: Default domain type: Passthrough [ 0.761386] SCSI subsystem initialized [ 0.764155] ACPI: bus type USB registered [ 0.765088] usbcore: registered new interface driver usbfs [ 0.767075] usbcore: registered new interface driver hub [ 0.769085] usbcore: registered new device driver usb [ 0.770163] pps_core: LinuxPPS API ver. 1 registered [ 0.772012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.775061] PTP clock support registered [ 0.777203] EDAC MC: Ver: 3.0.0 [ 0.779137] PCI: Using ACPI for IRQ routing [ 0.780859] NetLabel: Initializing [ 0.782010] NetLabel: domain hash size = 128 [ 0.784017] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.786083] NetLabel: unlabeled traffic allowed by default [ 0.788159] vgaarb: loaded [ 0.790297] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.792016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.797338] clocksource: Switched to clocksource kvm-clock [ 0.906627] VFS: Disk quotas dquot_6.6.0 [ 0.908495] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.911255] *** VALIDATE ramfs *** [ 0.912768] *** VALIDATE hugetlbfs *** [ 0.914535] pnp: PnP ACPI init [ 0.917256] pnp: PnP ACPI: found 6 devices [ 0.938328] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.942481] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.945053] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.947647] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.950384] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.953304] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.956386] NET: Registered protocol family 2 [ 0.959097] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.964082] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.967775] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.973894] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.977541] TCP: Hash tables configured (established 65536 bind 65536) [ 0.980782] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.984208] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.987298] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.990524] NET: Registered protocol family 1 [ 0.994165] RPC: Registered named UNIX socket transport module. [ 0.996485] RPC: Registered udp transport module. [ 0.998361] RPC: Registered tcp transport module. [ 1.000207] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.002846] NET: Registered protocol family 44 [ 1.004576] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.006912] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.009122] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.013382] PCI: CLS 0 bytes, default 64 [ 1.015052] Unpacking initramfs... [ 2.413099] debug: unmapping init [mem 0xffff9fd0fcc54000-0xffff9fd0fffbffff] [ 2.416659] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.418312] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.420371] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.933378] Initialise system trusted keyrings [ 2.935743] Key type blacklist registered [ 2.938516] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.946847] zbud: loaded [ 2.949951] *** VALIDATE nfs *** [ 2.950905] *** VALIDATE nfs4 *** [ 2.952426] pstore: using deflate compression [ 2.956310] Platform Keyring initialized [ 3.071893] NET: Registered protocol family 38 [ 3.074244] Key type asymmetric registered [ 3.076343] Asymmetric key parser 'x509' registered [ 3.078918] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.083443] io scheduler mq-deadline registered [ 3.085453] io scheduler kyber registered [ 3.087521] io scheduler bfq registered [ 3.090039] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.107224] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.110921] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.115582] ACPI: Power Button [PWRF] [ 3.124418] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.132406] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.147560] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.156895] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.189853] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.219522] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.248022] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.255619] Non-volatile memory driver v1.3 [ 3.257196] Linux agpgart interface v0.103 [ 3.291879] virtio_blk virtio1: [vda] 146152 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.294354] vda: detected capacity change from 0 to 74829824 [ 3.307906] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.310364] vdb: detected capacity change from 0 to 1073741824 [ 3.324164] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.326581] vdc: detected capacity change from 0 to 2621440000 [ 3.338703] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.341211] vdd: detected capacity change from 0 to 2621440000 [ 3.354671] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.357256] vde: detected capacity change from 0 to 4294967296 [ 3.373495] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.377710] vdf: detected capacity change from 0 to 4294967296 [ 3.390958] libphy: Fixed MDIO Bus: probed [ 3.398101] usbcore: registered new interface driver usbserial_generic [ 3.400834] usbserial: USB Serial support registered for generic [ 3.403161] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.407646] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.409326] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.411903] mousedev: PS/2 mouse device common for all mice [ 3.415556] rtc_cmos 00:05: RTC can wake from S4 [ 3.418738] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.420087] rtc_cmos 00:05: registered as rtc0 [ 3.423975] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.425463] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.426898] intel_pstate: CPU model not supported [ 3.431247] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.435455] hid: raw HID events driver (C) Jiri Kosina [ 3.438148] usbcore: registered new interface driver usbhid [ 3.439925] usbhid: USB HID core driver [ 3.441833] drop_monitor: Initializing network drop monitor service [ 3.444521] Initializing XFRM netlink socket [ 3.446826] NET: Registered protocol family 10 [ 3.449933] Segment Routing with IPv6 [ 3.451641] NET: Registered protocol family 17 [ 3.454134] mpls_gso: MPLS GSO support [ 3.458582] RAS: Correctable Errors collector initialized. [ 3.460366] AVX version of gcm_enc/dec engaged. [ 3.462217] AES CTR mode by8 optimization enabled [ 3.547396] sched_clock: Marking stable (3547370063, 0)->(4495358760, -947988697) [ 3.552403] registered taskstats version 1 [ 3.554709] Loading compiled-in X.509 certificates [ 3.556727] zswap: loaded using pool lzo/zbud [ 3.581551] Key type big_key registered [ 3.594474] Key type encrypted registered [ 3.596291] ima: No TPM chip found, activating TPM-bypass! [ 3.598461] ima: Allocated hash algorithm: sha1 [ 3.600404] ima: No architecture policies found [ 3.602348] evm: Initialising EVM extended attributes: [ 3.604423] evm: security.selinux [ 3.605884] evm: security.ima [ 3.606975] evm: security.capability [ 3.608122] evm: HMAC attrs: 0x1 [ 3.610163] rtc_cmos 00:05: setting system clock to 2026-08-24 22:46:35 UTC (1787611595) [ 3.616527] debug: unmapping init [mem 0xffffffffb5603000-0xffffffffb57fffff] [ 3.619975] debug: unmapping init [mem 0xffffffffb4382000-0xffffffffb4658fff] [ 3.627225] Write protecting the kernel read-only data: 28672k [ 3.630444] debug: unmapping init [mem 0xffffffffb2a03000-0xffffffffb2bfffff] [ 3.633591] debug: unmapping init [mem 0xffffffffb3314000-0xffffffffb33fffff] [ 3.671496] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.679979] systemd[1]: Detected virtualization kvm. [ 3.681681] systemd[1]: Detected architecture x86-64. [ 3.684049] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.708693] systemd[1]: No hostname configured. [ 3.710372] systemd[1]: Set hostname to . [ 3.713683] random: systemd: uninitialized urandom read (16 bytes read) [ 3.716834] systemd[1]: Initializing machine ID from random generator. [ 3.763901] random: ln: uninitialized urandom read (6 bytes read) [ 3.863817] random: systemd: uninitialized urandom read (16 bytes read) [ 3.866626] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.870621] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.883230] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.559522] device-mapper: uevent: version 1.0.3 [ 4.562417] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.319175] random: fast init done [ 5.322230] virtio_net virtio0 ens2: renamed from eth0 [ 5.388931] scsi host0: ata_piix [ 5.426156] scsi host1: ata_piix [ 5.427595] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.430394] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.974379] dracut-initqueue[582]: RTNETLINK answers: File exists [ 10.112122] random: crng init done [ 10.113655] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.563787] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.846412] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.148338] SELinux: Disabled at runtime. [ 12.218885] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.229201] systemd[1]: Detected virtualization kvm. [ 12.231074] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.870904] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.875197] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.882876] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.886157] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.888657] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.896325] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.900987] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... [ OK ] Reached target Slices. Mounting Huge Pages File System... Mounting POSIX Message Queue File Syste[ 13.048588] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS m... Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.433588] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.701263] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.756342] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.852564] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.878592] EDAC sbridge: Ver: 1.1.2 [ 15.645073] Key type dns_resolver registered [ 15.969519] NFS: Registering the id_resolver key type [ 15.972145] Key type id_resolver registered [ 15.974814] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg628-server login: [ 30.110111] hrtimer: interrupt took 3102170 ns [ 79.927431] libcfs: loading out-of-tree module taints kernel. [ 79.983994] Key type ._llcrypt registered [ 79.985852] Key type .llcrypt registered [ 80.080442] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [ 97.081848] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 98.499588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 98.512281] alg: No test for adler32 (adler32-zlib) [ 99.884521] Lustre: Lustre: Build Version: 2.17.57_80_g20f51fc [ 100.734221] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 102.471269] Key type lgssc registered [ 103.971077] Lustre: Echo OBD driver; http://www.lustre.org/ [ 120.608995] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 160.420581] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 172.721628] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 172.787825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 174.031793] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 174.064859] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 174.163232] Lustre: lustre-MDT0000: new disk, initializing [ 174.253740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 174.275976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 178.456260] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 190.812346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 190.905433] Lustre: 6509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 190.939630] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 190.945718] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 190.945954] Lustre: Skipped 1 previous similar message [ 191.006293] Lustre: lustre-MDT0001: new disk, initializing [ 191.047893] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 191.064691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 191.076765] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 195.544105] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 200.237609] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 209.282615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 209.581345] Lustre: lustre-OST0000: new disk, initializing [ 209.587025] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 209.593633] Lustre: 8446:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 209.701310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 216.307906] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 218.145405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 218.169384] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 218.220385] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 229.466535] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 229.603509] Lustre: lustre-OST0001: new disk, initializing [ 229.612942] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 229.626278] Lustre: 9519:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 229.688916] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 235.544530] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 236.636663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 236.645279] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 236.674884] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 246.098581] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 252.539145] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 258.740562] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing check_logdir /tmp/testlogs/ [ 263.595446] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing yml_node [ 269.503858] Lustre: DEBUG MARKER: Client: 2.17.57.80 [ 271.902340] Lustre: DEBUG MARKER: MDS: 2.17.57.80 [ 274.870371] Lustre: DEBUG MARKER: OSS: 2.17.57.80 [ 276.817660] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 24 18:51:06 EDT 2026 [ 294.396396] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 295.818862] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 313.823960] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 313.827072] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 313.831379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 313.846146] Lustre: Skipped 2 previous similar messages [ 317.371236] Lustre: server umount lustre-MDT0000 complete [ 320.869048] LustreError: 6503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787611912 with bad export cookie 11652430474382152765 [ 320.876031] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 320.877592] LustreError: 6503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 324.065605] LustreError: 10212:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 324.065842] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 324.066322] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 324.066326] Lustre: Skipped 3 previous similar messages [ 324.090181] LustreError: 10212:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 324.118904] Lustre: Skipped 2 previous similar messages [ 325.090123] LustreError: 6517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 325.113651] LustreError: 6517:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 330.216515] LustreError: 6518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 330.217564] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 330.231859] LustreError: 6518:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 330.244539] Lustre: Skipped 4 previous similar messages [ 335.328791] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 335.329034] LustreError: 6516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 335.329098] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 335.371389] LustreError: 6516:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 335.764816] Lustre: server umount lustre-MDT0001 complete [ 339.060180] Lustre: server umount lustre-OST0000 complete [ 342.195355] Lustre: server umount lustre-OST0001 complete [ 348.282822] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [ 356.753963] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 390.296586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 390.507335] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 390.527709] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 390.590786] Lustre: lustre-MDT0000: new disk, initializing [ 390.673640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 390.695385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 394.719169] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 405.216983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 405.368631] Lustre: 15072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 405.460685] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 405.465982] Lustre: Skipped 1 previous similar message [ 405.572530] Lustre: lustre-MDT0001: new disk, initializing [ 405.679274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 405.731680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 405.742084] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 409.617982] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 419.229643] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 425.206746] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 432.352328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 432.633569] Lustre: lustre-OST0000: new disk, initializing [ 432.638984] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 432.646852] Lustre: 17041:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 432.727344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 434.168114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 434.182937] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 434.309144] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 438.755965] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 448.150318] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 452.686211] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 452.880717] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 456.896860] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 457.065476] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 459.745773] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 459.747678] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 459.758355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 459.758363] Lustre: Skipped 1 previous similar message [ 465.158152] Lustre: server umount lustre-OST0000 complete [ 469.992645] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 470.018566] Lustre: Skipped 2 previous similar messages [ 475.397794] Lustre: server umount lustre-MDT0000 complete [ 478.423060] LustreError: 15066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787612070 with bad export cookie 11652430474382154298 [ 478.428657] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 478.434227] LustreError: 15066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 478.813878] Lustre: server umount lustre-MDT0001 complete [ 484.803245] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 18:54:34 (1787612074) [ 502.651617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 503.098229] LustreError: 19916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 503.167292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 507.148406] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 513.512314] LustreError: 19916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 513.533486] LustreError: 19916:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 516.731559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 521.965624] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 531.067762] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 537.254632] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 539.057885] Lustre: Failing over lustre-MDT0000 [ 539.354056] Lustre: server umount lustre-MDT0000 complete [ 541.155759] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 541.175597] LustreError: 19911:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 541.189374] LustreError: 19911:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 557.537362] Lustre: 3642:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787612133/real 1787612133] req@ffff9fd17f61c380 x1874446717706880/t0(0) o400->MGC192.168.206.128@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787612149 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 557.562427] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 558.653413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 568.808190] LustreError: 3641:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9fd043509f80 x1874446717710080/t0(0) o250->MGC192.168.206.128@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 569.092702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 569.101538] Lustre: Skipped 1 previous similar message [ 574.438377] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 574.447392] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 574.500453] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 574.667341] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 585.015408] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 589.792633] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 589.804238] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 589.822595] Lustre: Skipped 1 previous similar message [ 589.832900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 589.841360] Lustre: Skipped 7 previous similar messages [ 595.308352] Lustre: server umount lustre-MDT0000 complete [ 596.964811] LustreError: 20644:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 596.995955] LustreError: 20644:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 599.475539] LustreError: 21485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787612191 with bad export cookie 11652430474382155712 [ 599.480220] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 599.480716] LustreError: 21485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 599.687648] Lustre: server umount lustre-MDT0001 complete [ 608.911554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 613.761602] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 623.807839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 624.033941] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 624.041906] Lustre: Skipped 2 previous similar messages [ 629.501536] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 636.980539] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 642.985250] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 645.555793] Lustre: Failing over lustre-MDT0000 [ 646.087216] Lustre: server umount lustre-MDT0000 complete [ 649.698860] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 649.699223] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 649.727664] Lustre: Skipped 1 previous similar message [ 665.061258] LustreError: 23251:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 665.091638] LustreError: 23251:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 665.952153] Lustre: 3642:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787612241/real 1787612241] req@ffff9fd1805cdf80 x1874446717737216/t0(0) o400->MGC192.168.206.128@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787612257 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 666.015262] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 667.754468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 675.297863] LustreError: 3641:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9fd17f762d80 x1874446717740288/t0(0) o250->MGC192.168.206.128@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 675.624934] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 680.603908] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 680.935413] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 680.942691] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 680.960717] Lustre: Skipped 1 previous similar message [ 680.989913] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 690.369694] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 695.270618] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 695.284640] Lustre: Skipped 1 previous similar message [ 695.287981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 695.291817] Lustre: Skipped 1 previous similar message [ 700.709997] Lustre: server umount lustre-MDT0000 complete [ 704.171637] LustreError: 23234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787612296 with bad export cookie 11652430474382156650 [ 704.173233] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 704.181039] LustreError: 23234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 704.447642] Lustre: server umount lustre-MDT0001 complete [ 713.401524] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 18:58:22 (1787612302) [ 730.272825] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [ 740.952944] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 783.111696] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 793.854741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 794.076666] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 794.127399] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 794.272353] Lustre: lustre-MDT0000: new disk, initializing [ 794.345633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 794.350163] Lustre: Skipped 3 previous similar messages [ 794.364614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 798.662543] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 808.862615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 809.001569] Lustre: 29629:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 809.244497] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 813.015567] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 822.623805] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 827.891325] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 834.640251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 834.924945] Lustre: 31598:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 836.330780] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 836.379492] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 836.486302] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 836.504869] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 840.861265] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 850.100602] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 855.349283] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 877.545061] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 877.550909] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 877.551743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 877.551749] Lustre: Skipped 2 previous similar messages [ 877.567191] Lustre: Skipped 2 previous similar messages [ 887.265464] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 887.642704] Lustre: server umount lustre-OST0000 complete [ 898.020058] LustreError: 29643:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 898.033625] Lustre: server umount lustre-MDT0000 complete [ 898.038049] LustreError: 29643:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 901.569645] LustreError: 29623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787612493 with bad export cookie 11652430474382157056 [ 901.579542] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 901.587209] LustreError: 29623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 902.084928] Lustre: server umount lustre-MDT0001 complete [ 912.702124] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 915.653992] Key type lgssc unregistered [ 915.979517] LNet: 33592:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 915.983898] LNetError: 33592:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 915.996937] LNet: Removed LNI 192.168.206.128@tcp [ 916.757247] Key type .llcrypt unregistered [ 916.760203] Key type ._llcrypt unregistered [ 936.579874] Key type ._llcrypt registered [ 936.585472] Key type .llcrypt registered [ 936.659275] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 937.433370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 937.482941] alg: No test for adler32 (adler32-zlib) [ 938.519262] Lustre: Lustre: Build Version: 2.17.57_80_g20f51fc [ 938.722097] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 940.367265] Key type lgssc registered [ 941.407922] Lustre: Echo OBD driver; http://www.lustre.org/ [ 948.415763] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 948.433190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 949.813371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 954.858069] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 959.970458] LustreError: 35010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 959.985454] LustreError: 35010:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 963.556516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 963.854718] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 968.271655] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 976.801906] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 983.381582] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 990.266532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 990.569315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 994.720204] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 996.836340] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1003.461531] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1009.011247] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1020.531042] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 19:03:29 (1787612609) [ 1022.480128] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1024.626369] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1031.575711] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 19:03:40 (1787612620) [ 1040.604874] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 19:03:49 (1787612629) [ 1041.410716] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1048.986915] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1048.995193] Lustre: Skipped 2 previous similar messages [ 1050.778766] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 19:04:00 (1787612640) [ 1054.239847] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1054.251088] Lustre: Skipped 3 previous similar messages [ 1070.470827] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1070.480219] Lustre: Skipped 6 previous similar messages [ 1073.225838] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 19:04:22 (1787612662) [ 1075.774130] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1079.208893] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1088.414750] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 19:04:37 (1787612677) [ 1090.334106] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1090.337987] Lustre: Skipped 3 previous similar messages [ 1103.045930] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 19:04:52 (1787612692) [ 1116.641879] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1116.669149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1116.673519] Lustre: Skipped 1 previous similar message [ 1121.766294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1122.324771] Lustre: server umount lustre-OST0000 complete [ 1126.882562] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1126.887731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1126.893525] Lustre: Skipped 2 previous similar messages [ 1126.901295] Lustre: Skipped 1 previous similar message [ 1132.001489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1132.018335] Lustre: Skipped 2 previous similar messages [ 1132.845613] Lustre: server umount lustre-MDT0000 complete [ 1137.098876] LustreError: 34991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787612728 with bad export cookie 11750412542437397425 [ 1137.101262] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1137.112517] LustreError: 34991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1137.489601] Lustre: server umount lustre-MDT0001 complete [ 1149.703915] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 1152.589618] Key type lgssc unregistered [ 1152.933945] LNet: 42857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1152.940829] LNetError: 42857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1152.955653] LNet: Removed LNI 192.168.206.128@tcp [ 1153.916706] Key type .llcrypt unregistered [ 1153.920641] Key type ._llcrypt unregistered [ 1170.892615] Key type ._llcrypt registered [ 1170.896044] Key type .llcrypt registered [ 1171.026314] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [ 1185.493429] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 1186.729119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1186.795723] alg: No test for adler32 (adler32-zlib) [ 1187.860666] Lustre: Lustre: Build Version: 2.17.57_80_g20f51fc [ 1188.129962] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 1189.775161] Key type lgssc registered [ 1191.174822] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1232.449462] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1232.473306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1233.708422] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1233.738574] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1233.810321] Lustre: lustre-MDT0000: new disk, initializing [ 1233.863204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1233.875323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1238.348308] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1248.976909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1249.169243] Lustre: 46020:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1249.269575] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1249.280212] Lustre: Skipped 1 previous similar message [ 1249.394541] Lustre: lustre-MDT0001: new disk, initializing [ 1249.574920] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1249.640414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1249.668617] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1254.896492] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1266.927468] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1272.773411] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1280.259757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1280.540216] Lustre: lustre-OST0000: new disk, initializing [ 1280.545396] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1280.554597] Lustre: 47986:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1280.648515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1286.423227] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1290.333311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1290.345854] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1290.484596] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1294.893083] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1299.154837] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1299.370944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1304.587762] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1304.876664] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1311.205573] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1311.214032] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1311.223286] LustreError: Skipped 1 previous similar message [ 1311.232120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1314.304326] Lustre: server umount lustre-OST0000 complete [ 1321.440375] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1321.446621] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1321.480581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1321.504533] Lustre: Skipped 2 previous similar messages [ 1321.531118] Lustre: Skipped 2 previous similar messages [ 1326.561197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1326.568506] Lustre: Skipped 1 previous similar message [ 1331.682048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1331.687773] Lustre: Skipped 1 previous similar message [ 1334.240225] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1334.462281] Lustre: server umount lustre-MDT0000 complete [ 1339.911357] LustreError: 46012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787612931 with bad export cookie 11600077826922211942 [ 1339.930595] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1339.941072] LustreError: 46012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1340.504535] Lustre: server umount lustre-MDT0001 complete [ 1352.656536] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 19:09:01 (1787612941) [ 1359.687202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1359.971556] LustreError: 50450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1359.991371] LustreError: 50450:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1360.041780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1364.404767] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1365.473611] LustreError: 50451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1365.486537] LustreError: 50451:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1369.575282] LustreError: 50450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1372.287687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1372.744406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1377.159216] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1385.369911] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1393.073340] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1401.559407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1401.855972] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1407.464876] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1414.826235] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1422.900717] LustreError: 52978:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1423.918706] Lustre: Modifying parameter general.timeout=1 in log params [ 1425.844990] Lustre: Modifying parameter general.timeout=3 in log params [ 1425.849579] Lustre: Skipped 1 previous similar message [ 1427.923796] Lustre: Modifying parameter general.timeout=5 in log params [ 1427.941807] Lustre: Skipped 1 previous similar message [ 1432.157808] Lustre: Modifying parameter general.timeout=8 in log params [ 1432.169982] Lustre: Skipped 2 previous similar messages [ 1440.912456] Lustre: Modifying parameter general.timeout=17 in log params [ 1440.916497] Lustre: Skipped 8 previous similar messages [ 1457.133482] Lustre: Modifying parameter general.timeout=37 in log params [ 1457.137471] Lustre: Skipped 19 previous similar messages [ 1489.141515] Lustre: Modifying parameter general.timeout=76 in log params [ 1489.147817] Lustre: Skipped 38 previous similar messages [ 1550.158859] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 19:12:19 (1787613139) [ 1553.542567] Lustre: 59927:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1554.500487] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 in log params [ 1554.507522] Lustre: Skipped 56 previous similar messages [ 1555.522675] Lustre: 60025:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1559.284947] Lustre: 60235:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1559.295191] Lustre: 60235:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1591.488325] Lustre: 61808:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1591.498627] Lustre: 61808:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1591.522178] LustreError: 61808:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1596.090149] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 19:13:05 (1787613185) [ 1613.281974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1613.289131] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1613.297802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1613.316831] Lustre: Skipped 2 previous similar messages [ 1615.800903] Lustre: server umount lustre-MDT0000 complete [ 1618.401386] LustreError: 50451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1618.418464] LustreError: 50451:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1620.132733] LustreError: 50431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787613212 with bad export cookie 11600077826922212824 [ 1620.146543] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1620.149900] LustreError: 50431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1620.523172] Lustre: server umount lustre-MDT0001 complete [ 1634.475836] Lustre: server umount lustre-OST0000 complete [ 1663.490206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1663.782868] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1663.997142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1667.972188] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1675.695954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1675.861329] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1675.930551] Lustre: 64340:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1680.581244] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1688.079894] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1694.628342] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1701.326446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1701.492511] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1701.822754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1701.829788] Lustre: Skipped 1 previous similar message [ 1709.136695] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1716.631845] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1718.451730] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1718.475942] Lustre: Skipped 20 previous similar messages [ 1718.525727] Lustre: 66906:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1728.997656] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1729.007234] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1729.026346] Lustre: Skipped 2 previous similar messages [ 1729.034995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1731.207143] Lustre: server umount lustre-OST0000 complete [ 1734.626859] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1734.639982] Lustre: Skipped 2 previous similar messages [ 1740.859260] Lustre: server umount lustre-MDT0000 complete [ 1744.115969] LustreError: 64333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787613336 with bad export cookie 11600077826922215883 [ 1744.128480] LustreError: 64333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1744.129080] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1744.495933] Lustre: server umount lustre-MDT0001 complete [ 1755.088499] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 1757.863545] Key type lgssc unregistered [ 1758.130283] LNet: 68116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1758.138879] LNetError: 68116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1758.158822] LNet: Removed LNI 192.168.206.128@tcp [ 1758.856850] Key type .llcrypt unregistered [ 1758.860882] Key type ._llcrypt unregistered [ 1780.942125] Key type ._llcrypt registered [ 1780.946871] Key type .llcrypt registered [ 1781.025920] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 1782.466154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1782.512077] alg: No test for adler32 (adler32-zlib) [ 1783.569387] Lustre: Lustre: Build Version: 2.17.57_80_g20f51fc [ 1783.889550] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 1785.640937] Key type lgssc registered [ 1786.888500] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1797.183747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1797.199272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1798.807176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1804.110551] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1808.353894] LustreError: 69530:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1808.385895] LustreError: 69530:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1813.126031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1813.776161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1818.689747] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1826.404642] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1832.471834] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1838.551925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1838.977927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1844.537849] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1851.698925] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1863.457547] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1873.902675] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1873.910246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1873.920241] Lustre: Skipped 2 previous similar messages [ 1873.926870] Lustre: Skipped 1 previous similar message [ 1878.438316] LustreError: 72575:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000dcc2daa1) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 1878.745673] Lustre: server umount lustre-MDT0000 complete [ 1879.009823] LustreError: 69525:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1879.030307] LustreError: 69525:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1882.454994] LustreError: 69511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787613474 with bad export cookie 13633910341230577814 [ 1882.469450] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1882.474052] LustreError: 69511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1882.896676] Lustre: server umount lustre-MDT0001 complete [ 1897.282797] Lustre: server umount lustre-OST0000 complete [ 1930.861843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1931.204952] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1931.251495] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1931.792989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1936.526857] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1947.685723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1947.793964] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1947.827500] Lustre: 74405:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1948.245940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1953.883708] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1961.267524] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1966.736914] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1974.374981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1974.657519] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1975.128073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1981.737040] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1989.889162] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1991.092938] Lustre: 76979:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1993.812565] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2003.943457] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2003.961803] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2003.976119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2003.986213] Lustre: Skipped 1 previous similar message [ 2004.454356] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2007.677791] Lustre: server umount lustre-OST0000 complete [ 2015.104382] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2015.153362] Lustre: Skipped 1 previous similar message [ 2015.193092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2015.205705] Lustre: Skipped 3 previous similar messages [ 2020.321309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2020.340200] Lustre: Skipped 1 previous similar message [ 2022.762219] Lustre: server umount lustre-MDT0000 complete [ 2025.443119] LustreError: 75490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2025.509660] LustreError: 75490:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2029.670805] LustreError: 74416:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2029.702759] LustreError: 74416:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2031.948592] LustreError: 74396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787613623 with bad export cookie 13633910341230578654 [ 2031.958254] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2031.973647] LustreError: 74396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2032.757889] Lustre: server umount lustre-MDT0001 complete [ 2056.435138] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 2059.815127] Key type lgssc unregistered [ 2060.260191] LNet: 78228:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2060.273635] LNetError: 78228:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2060.318171] LNet: Removed LNI 192.168.206.128@tcp [ 2061.535278] Key type .llcrypt unregistered [ 2061.537453] Key type ._llcrypt unregistered [ 2096.106042] Key type ._llcrypt registered [ 2096.110094] Key type .llcrypt registered [ 2096.230704] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 2097.505917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2097.643950] alg: No test for adler32 (adler32-zlib) [ 2098.855310] Lustre: Lustre: Build Version: 2.17.57_80_g20f51fc [ 2099.095715] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 2100.791537] Key type lgssc registered [ 2102.317070] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2111.363518] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2111.375251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2112.975773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2117.642110] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2122.208889] LustreError: 79643:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2122.227411] LustreError: 79643:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2126.126867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2126.885861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2131.416546] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2138.556449] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2144.631632] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2151.257470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2151.341873] LustreError: 81471:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2151.977197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2158.391626] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2166.179325] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2173.341259] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2174.339628] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2176.083220] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2176.087546] Lustre: Skipped 1 previous similar message [ 2178.183210] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2178.187616] Lustre: Skipped 1 previous similar message [ 2183.134566] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2183.140106] Lustre: Skipped 4 previous similar messages [ 2192.057418] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2192.064194] Lustre: Skipped 8 previous similar messages [ 2208.726423] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2208.730624] Lustre: Skipped 16 previous similar messages [ 2241.613854] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2241.626850] Lustre: Skipped 34 previous similar messages [ 2276.186700] Lustre: 87741:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2278.709585] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 19:24:28 (1787613868) [ 2280.327693] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2282.447694] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 19:24:31 (1787613871) [ 2290.145165] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2290.154819] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2290.183172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2291.173215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2291.175257] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2291.200594] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2293.713811] Lustre: server umount lustre-OST0000 complete [ 2300.899570] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2300.907827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2300.929295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2300.935379] Lustre: Skipped 1 previous similar message [ 2304.719549] Lustre: server umount lustre-MDT0000 complete [ 2306.531610] LustreError: 79639:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2306.548317] LustreError: 79639:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2308.473085] LustreError: 79624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787613900 with bad export cookie 13653780397488181428 [ 2308.475983] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2308.481691] LustreError: 79624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2308.872792] Lustre: server umount lustre-MDT0001 complete [ 2319.993403] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 2323.648185] Key type lgssc unregistered [ 2324.076548] LNet: 88993:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2324.089409] LNetError: 88993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2324.104780] LNet: Removed LNI 192.168.206.128@tcp [ 2325.493314] Key type .llcrypt unregistered [ 2325.497968] Key type ._llcrypt unregistered [ 2332.761460] Key type ._llcrypt registered [ 2332.763141] Key type .llcrypt registered [ 2332.954325] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2339.962525] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules [ 2341.792637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2341.888351] alg: No test for adler32 (adler32-zlib) [ 2343.820022] Lustre: Lustre: Build Version: 2.17.57_80_g20f51fc [ 2344.426054] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 2346.176316] Key type lgssc registered [ 2347.879394] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2349.743321] LustreError: 90245:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2368.112925] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 2372.825060] LustreError: 90245:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2379.592376] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2379.611561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2381.323689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2386.963800] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2391.529430] LustreError: 91274:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2391.556325] LustreError: 91274:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2396.647815] LustreError: 91275:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2398.860270] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 19:26:27 (1787613987) [ 2401.765562] LustreError: 91274:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2404.696299] LustreError: 91306:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2404.708575] LustreError: 91306:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2404.715466] LustreError: 91306:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 23, retries 0, failed: rc = -5 [ 2405.120548] Lustre: server umount lustre-MDT0000 complete [ 2421.628234] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 2425.970314] Key type lgssc unregistered [ 2426.397691] LNet: 92802:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2426.408547] LNetError: 92802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2426.432362] LNet: Removed LNI 192.168.206.128@tcp [ 2427.455307] Key type .llcrypt unregistered [ 2427.462427] Key type ._llcrypt unregistered [ 2448.240165] Key type ._llcrypt registered [ 2448.246399] Key type .llcrypt registered [ 2448.369542] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 2449.678686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2449.718493] alg: No test for adler32 (adler32-zlib) [ 2450.761789] Lustre: Lustre: Build Version: 2.17.57_80_g20f51fc [ 2451.047686] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 2452.737458] Key type lgssc registered [ 2453.987870] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2462.827454] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2462.865503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2464.554263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2469.776795] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2474.988863] LustreError: 94024:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2475.021905] LustreError: 94024:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2479.601826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2480.379653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2485.563932] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2494.069453] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2500.877709] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2509.228133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2509.344566] LustreError: 95851:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2509.951208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2517.157122] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2525.485634] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2579.210688] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 19:29:28 (1787614168) [ 2581.445241] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2583.465586] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 19:29:32 (1787614172) [ 2591.205308] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2591.223381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2591.226150] Lustre: Skipped 2 previous similar messages [ 2591.232059] Lustre: Skipped 1 previous similar message [ 2595.462131] Lustre: server umount lustre-MDT0000 complete [ 2596.323000] LustreError: 94024:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2596.362806] LustreError: 94024:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2599.725574] LustreError: 94003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787614191 with bad export cookie 183724986914727802 [ 2599.728346] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2599.740210] LustreError: 94003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2600.351943] Lustre: server umount lustre-MDT0001 complete [ 2615.804291] Lustre: server umount lustre-OST0000 complete [ 2627.988706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2628.551410] LustreError: 98320:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2628.740538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2633.919714] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2634.210266] LustreError: 98321:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2642.715682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2643.307199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2648.665967] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2656.772826] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2663.446442] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2677.582740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2677.928296] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2677.935936] LustreError: 98309:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2677.963960] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2677.983908] LustreError: 100403:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2677.993558] LustreError: 100403:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2678.001442] LustreError: 100403:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2678.204061] Lustre: server umount lustre-OST0000 complete [ 2678.217540] LustreError: 100403:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2687.633103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2687.891577] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2687.910862] LustreError: 98308:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2687.961926] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2687.977633] LustreError: 100869:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2687.991512] LustreError: 100869:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2688.000353] LustreError: 100869:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2688.102605] Lustre: server umount lustre-OST0000 complete [ 2688.107112] LustreError: 100869:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2699.990817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2700.183669] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2700.190531] Lustre: Found index 0 for lustre-OST0000, updating log [ 2700.196540] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2700.259414] Lustre: lustre-OST0000: new disk, initializing [ 2700.264798] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2700.272346] Lustre: 101409:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2700.518306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2707.877718] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2712.071177] Lustre: Failing over lustre-OST0000 [ 2712.220136] Lustre: server umount lustre-OST0000 complete [ 2718.179986] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2718.184151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2718.185353] Lustre: Skipped 1 previous similar message [ 2718.229247] Lustre: Skipped 2 previous similar messages [ 2722.037456] Lustre: server umount lustre-MDT0000 complete [ 2723.296209] LustreError: 98321:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2723.330961] LustreError: 98321:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2725.866308] LustreError: 99061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787614317 with bad export cookie 183724986914730546 [ 2725.870810] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2725.880035] LustreError: 99061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2726.202541] Lustre: server umount lustre-MDT0001 complete [ 2735.126321] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 19:32:04 (1787614324) [ 2752.243571] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 2763.560987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2764.143564] LustreError: 103753:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2764.326632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2768.991621] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2777.712479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2782.792419] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2785.563295] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2792.389627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2799.574757] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2808.836136] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2808.948398] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2809.070338] Lustre: lustre-OST0001: new disk, initializing [ 2809.073414] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2809.079064] Lustre: 106070:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2809.391306] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2809.394880] Lustre: Skipped 2 previous similar messages [ 2816.236426] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2818.192820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2818.209221] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2818.311281] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2823.923178] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2829.588619] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2833.890529] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2833.897983] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2833.910372] Lustre: Skipped 1 previous similar message [ 2833.923401] Lustre: Skipped 2 previous similar messages [ 2839.026888] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2839.032616] Lustre: Skipped 3 previous similar messages [ 2839.743493] Lustre: server umount lustre-MDT0001 complete [ 2844.133169] LustreError: 103750:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2844.148335] LustreError: 103750:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2849.239650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2849.396671] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2849.410689] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2849.514220] Lustre: 103743:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2850.779053] LustreError: 107603:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2850.791190] LustreError: 107603:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2850.812252] LustreError: 103745:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2850.922396] LustreError: 107613:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.128@tcp: cfg command failed: rc = -17 [ 2850.934229] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2850.934229] [ 2855.403937] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2855.935196] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2866.503851] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 19:34:15 (1787614455) [ 2884.251486] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 2904.738750] Lustre: 109684:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2928.428870] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2932.483531] Lustre: 110820:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3018.724517] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3018.727618] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3018.734349] Lustre: Skipped 1 previous similar message [ 3018.737438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3018.752900] Lustre: Skipped 1 previous similar message [ 3023.069960] Lustre: server umount lustre-MDT0000 complete [ 3024.356747] LustreError: 103747:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3024.369107] LustreError: 103747:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 3027.222381] LustreError: 106789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787614619 with bad export cookie 183724986914731540 [ 3027.223743] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3027.235627] LustreError: 106789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3027.713703] Lustre: server umount lustre-MDT0001 complete [ 3041.261028] Lustre: server umount lustre-OST0000 complete [ 3054.554615] Lustre: server umount lustre-OST0001 complete [ 3060.138218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3071.046865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3081.261929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3091.125105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3110.980492] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 3121.388557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3121.411951] Lustre: lustre-MDT0000: reset Object Index mappings [ 3121.977460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3121.989416] Lustre: Skipped 1 previous similar message [ 3126.619180] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3134.633756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3134.658178] Lustre: lustre-MDT0001: reset Object Index mappings [ 3139.151774] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3142.070298] Lustre: 115924:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3148.400598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3155.265613] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3159.491826] LustreError: 116276:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3159.509254] LustreError: 116276:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3159.534196] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3164.401539] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3167.102399] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3167.103944] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3167.203196] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3171.066966] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3178.854857] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3182.924303] Lustre: 117801:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3236.858251] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 19:40:25 (1787614825) [ 3262.945118] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3262.965339] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3262.993187] Lustre: Skipped 3 previous similar messages [ 3263.012467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3263.018442] Lustre: Skipped 4 previous similar messages [ 3268.954960] Lustre: server umount lustre-MDT0000 complete [ 3272.498275] LustreError: 114756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787614864 with bad export cookie 183724986914778398 [ 3272.507372] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3272.510087] LustreError: 114756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3272.676452] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3272.687187] Lustre: Skipped 4 previous similar messages [ 3278.967756] Lustre: server umount lustre-MDT0001 complete [ 3282.863632] LustreError: 118557:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x280000401:0xf4:0x0].0x0 (ffff9fd174ab0400) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3297.248486] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3297.368605] Lustre: server umount lustre-OST0000 complete [ 3307.175855] Lustre: server umount lustre-OST0001 complete [ 3313.995243] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [ 3324.602821] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 3362.520768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3362.758567] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3362.790657] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3362.896292] Lustre: lustre-MDT0000: new disk, initializing [ 3362.994550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3363.003246] Lustre: Skipped 3 previous similar messages [ 3363.017622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3367.670932] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3374.803151] Lustre: server umount lustre-MDT0000 complete [ 3384.618874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3385.062163] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3385.089553] Lustre: 122069:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3385.125064] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3389.875764] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3394.871918] Lustre: server umount lustre-MDT0000 complete [ 3402.880944] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 19:43:12 (1787614992) [ 3404.350772] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3405.908298] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 19:43:15 (1787614995) [ 3424.523435] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 3435.854569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3440.530477] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3448.942161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3449.134711] Lustre: 124117:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3449.243914] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3449.256617] Lustre: Skipped 1 previous similar message [ 3449.426413] Lustre: lustre-MDT0001: new disk, initializing [ 3449.627756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3449.644129] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3455.607197] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3460.924470] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3467.615849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3467.885903] Lustre: lustre-OST0000: new disk, initializing [ 3467.890988] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3467.897990] Lustre: 125699:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3469.527907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3469.544908] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3469.633669] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3474.210453] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3483.562723] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3483.711496] Lustre: lustre-OST0001: new disk, initializing [ 3483.714651] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3483.720648] Lustre: 126569:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3485.182312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3485.196228] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3485.321055] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3489.522252] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3498.783038] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3502.769263] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3893.010478] Lustre: 132985:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3893.031322] Lustre: 132985:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3893.041418] Lustre: 132985:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3893.051897] Lustre: 132985:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3893.062186] Lustre: 132985:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3893.071977] Lustre: 132985:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4156.096796] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 19:55:45 (1787615745) [ 4171.744484] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4171.761254] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4171.777626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4171.782131] Lustre: Skipped 8 previous similar messages [ 4175.332516] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4175.338156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4175.351189] Lustre: Skipped 1 previous similar message [ 4175.358568] Lustre: Skipped 1 previous similar message [ 4177.377651] Lustre: server umount lustre-MDT0000 complete [ 4180.451480] LustreError: 124942:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4180.476022] LustreError: 124942:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4180.878103] LustreError: 124111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787615772 with bad export cookie 183724986914846949 [ 4180.885706] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4180.886050] LustreError: 124111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4185.571702] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4185.584517] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4185.589095] Lustre: Skipped 2 previous similar messages [ 4185.602121] Lustre: Skipped 3 previous similar messages [ 4195.295271] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4195.666909] Lustre: server umount lustre-MDT0001 complete [ 4214.243131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4214.457389] Lustre: server umount lustre-OST0000 complete [ 4224.314845] Lustre: server umount lustre-OST0001 complete [ 4230.635043] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [ 4240.819570] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 4276.089248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4276.347683] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4276.375472] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4276.477591] Lustre: lustre-MDT0000: new disk, initializing [ 4276.576861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4276.581202] Lustre: Skipped 5 previous similar messages [ 4276.590514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4281.185698] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 4293.150654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4293.330602] Lustre: 136036:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4293.393850] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4293.396691] Lustre: Skipped 1 previous similar message [ 4293.467241] Lustre: lustre-MDT0001: new disk, initializing [ 4293.615155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4293.631528] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4298.668049] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 4308.590457] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4313.715587] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4320.627434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4320.952552] Lustre: lustre-OST0000: new disk, initializing [ 4320.959627] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4320.970548] Lustre: 138005:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4321.123319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4321.139252] Lustre: Skipped 1 previous similar message [ 4322.317718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4322.333471] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4322.396689] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4327.320357] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 4337.250240] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4346.757336] Lustre: lustre-MDD0000: changelog on [ 4349.484467] Lustre: lustre-MDD0001: changelog on [ 6332.696180] Lustre: lustre-MDD0001: changelog off [ 6334.711438] Lustre: 136043:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6335.611186] Lustre: lustre-MDD0000: changelog off [ 6344.686220] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6344.700953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6344.706786] Lustre: Skipped 1 previous similar message [ 6344.711656] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6344.724074] Lustre: Skipped 3 previous similar messages [ 6349.123615] Lustre: server umount lustre-OST0000 complete [ 6354.912954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6354.930730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6359.933677] Lustre: server umount lustre-MDT0000 complete [ 6360.032921] LustreError: 136048:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6360.053725] LustreError: 136048:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6363.056246] LustreError: 136029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787617954 with bad export cookie 183724986915625979 [ 6363.064752] LustreError: 136029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6363.068561] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6363.415299] Lustre: server umount lustre-MDT0001 complete [ 6373.991569] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 6376.508428] Key type lgssc unregistered [ 6377.002996] LNet: 147176:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6377.015327] LNetError: 147176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6377.050379] LNet: Removed LNI 192.168.206.128@tcp [ 6378.284163] Key type .llcrypt unregistered [ 6378.286315] Key type ._llcrypt unregistered [ 6401.453044] Key type ._llcrypt registered [ 6401.455214] Key type .llcrypt registered [ 6401.550421] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [ 6414.753544] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 6415.568515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6415.734514] alg: No test for adler32 (adler32-zlib) [ 6416.812329] Lustre: Lustre: Build Version: 2.17.57_80_g20f51fc [ 6417.065967] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 6418.783192] Key type lgssc registered [ 6419.942527] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6458.250200] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6458.308204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6459.596328] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6459.624837] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6459.714440] Lustre: lustre-MDT0000: new disk, initializing [ 6459.769650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6459.788335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6464.846546] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6474.954777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6475.065639] Lustre: 150360:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6475.115810] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6475.119912] Lustre: Skipped 1 previous similar message [ 6475.172277] Lustre: lustre-MDT0001: new disk, initializing [ 6475.261963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6475.293255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6475.322334] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6479.457638] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6488.075680] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6493.274592] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6499.220745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6499.539315] Lustre: lustre-OST0000: new disk, initializing [ 6499.542500] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6499.547288] Lustre: 152328:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6499.618755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6505.271691] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6505.505280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6505.521858] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6505.631201] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6514.532596] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6520.739235] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_module obdecho/obdecho [ 6522.626490] LustreError: 153331:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6522.633150] LustreError: 153331:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6524.734187] LustreError: 153427:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6536.672733] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6536.680152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6536.686187] Lustre: Skipped 1 previous similar message [ 6536.689801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6536.695389] Lustre: Skipped 1 previous similar message [ 6541.794032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6541.798312] Lustre: Skipped 2 previous similar messages [ 6542.499434] Lustre: server umount lustre-MDT0000 complete [ 6545.586145] LustreError: 150353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787618137 with bad export cookie 9686016983052377228 [ 6545.588139] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6545.594588] LustreError: 150353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6545.893152] Lustre: server umount lustre-MDT0001 complete [ 6559.783409] Lustre: server umount lustre-OST0000 complete [ 6582.635771] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [ 6592.816601] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 6634.004501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6634.172796] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6634.191941] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6634.250660] Lustre: lustre-MDT0000: new disk, initializing [ 6634.302667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6634.322909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6639.541388] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6651.229244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6651.368674] Lustre: 157142:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6651.425296] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6651.429425] Lustre: Skipped 1 previous similar message [ 6651.508628] Lustre: lustre-MDT0001: new disk, initializing [ 6651.634591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6651.681705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6651.699231] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6657.144587] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6668.420364] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6673.906301] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6680.544330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6680.810793] Lustre: lustre-OST0000: new disk, initializing [ 6680.816783] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6680.824668] Lustre: 159115:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6680.934420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6686.775647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6686.793964] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6686.953458] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6686.958422] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6695.461844] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6699.041903] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6699.242319] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6703.484193] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6703.688540] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6707.681065] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6707.697589] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6707.703704] Lustre: Skipped 1 previous similar message [ 6707.707418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6707.712843] Lustre: Skipped 1 previous similar message [ 6711.523902] Lustre: server umount lustre-OST0000 complete [ 6717.931304] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6717.950874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6717.957428] Lustre: Skipped 2 previous similar messages [ 6717.981658] Lustre: Skipped 1 previous similar message [ 6721.372912] Lustre: server umount lustre-MDT0000 complete [ 6725.121348] LustreError: 157136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787618317 with bad export cookie 9686016983052378194 [ 6725.122833] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6725.131461] LustreError: 157136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6725.823457] Lustre: server umount lustre-MDT0001 complete [ 6729.387222] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 20:38:38 (1787618318) [ 6730.920474] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6732.730970] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 20:38:42 (1787618322) [ 6740.879834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6741.445757] LustreError: 161526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6741.463619] LustreError: 161526:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6741.537866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6746.397706] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6746.596212] LustreError: 161527:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6751.722508] LustreError: 161526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6755.592985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6760.838577] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6767.565600] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6772.143456] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6777.530128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6777.754739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6777.763744] Lustre: Skipped 1 previous similar message [ 6783.333400] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6789.883814] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6795.838577] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.28@tcp (stopping) [ 6796.768976] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6796.789659] LustreError: Skipped 1 previous similar message [ 6796.798238] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6802.142704] Lustre: server umount lustre-MDT0000 complete [ 6802.402930] LustreError: 161527:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6802.415836] LustreError: 161527:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6805.730429] LustreError: 163369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787618397 with bad export cookie 9686016983052379076 [ 6805.738182] LustreError: 163369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6805.742225] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6806.089205] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.28@tcp (stopping) [ 6806.099225] Lustre: Skipped 5 previous similar messages [ 6807.520995] LustreError: 162768:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6807.521498] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6807.550676] LustreError: 162768:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6807.582069] Lustre: Skipped 2 previous similar messages [ 6815.666071] LustreError: 162768:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6815.692133] LustreError: 162768:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6820.319129] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6820.563716] Lustre: server umount lustre-MDT0001 complete [ 6826.198320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6837.506130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6837.772441] LustreError: 165367:0:(llog.c:1655:llog_backup()) MGC192.168.206.128@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6837.777685] Lustre: 165367:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.128@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6845.417825] LustreError: 165372:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6845.444293] LustreError: 165372:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6878.519639] LustreError: 165373:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6878.584391] LustreError: 165373:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 27 previous similar messages [ 6898.655604] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6898.664666] Lustre: 165367:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.128@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6939.645419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6944.316915] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6944.737348] LustreError: 165374:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6944.739586] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6944.765037] LustreError: 165374:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 42 previous similar messages [ 6953.251796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6967.277636] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 6976.496036] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x866ba5209b3cfd01 [ 6976.509215] Lustre: MGC192.168.206.128@tcp: Connection restored to 0@lo (at 0@lo) [ 6980.618189] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6982.125711] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6987.949668] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6989.740900] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6995.702496] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6997.749936] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7017.955326] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7017.964195] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7017.968326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7017.968335] Lustre: Skipped 5 previous similar messages [ 7017.993262] Lustre: Skipped 2 previous similar messages [ 7021.984942] Lustre: server umount lustre-MDT0000 complete [ 7026.479428] LustreError: 161508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787618618 with bad export cookie 9686016983052385537 [ 7026.490259] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7026.499340] LustreError: 161508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7026.924261] Lustre: server umount lustre-MDT0001 complete [ 7041.358505] Lustre: server umount lustre-OST0000 complete [ 7062.369362] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [ 7071.234038] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 7105.251150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7105.447986] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7105.491265] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7105.575571] Lustre: lustre-MDT0000: new disk, initializing [ 7105.683410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7105.689367] Lustre: Skipped 1 previous similar message [ 7105.701169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7110.376879] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 7119.297337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7119.381465] Lustre: 170476:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7119.582060] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7122.709610] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 7130.487069] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7135.101942] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7141.779401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7142.080471] Lustre: lustre-OST0000: new disk, initializing [ 7142.089400] Lustre: Skipped 1 previous similar message [ 7142.113838] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7142.117409] Lustre: Skipped 2 previous similar messages [ 7142.122489] Lustre: 172447:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7148.820929] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 7150.685818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7150.703919] Lustre: Skipped 1 previous similar message [ 7150.714486] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7150.907966] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7159.443805] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7164.332587] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7164.582393] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7170.266456] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7170.576380] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7176.672278] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7176.680974] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7176.686933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7176.691927] Lustre: Skipped 1 previous similar message [ 7176.705863] Lustre: Skipped 3 previous similar messages [ 7179.437942] Lustre: server umount lustre-OST0000 complete [ 7197.151181] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7197.312722] Lustre: server umount lustre-MDT0000 complete [ 7201.135480] LustreError: 170471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787618793 with bad export cookie 9686016983052402358 [ 7201.140190] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7201.152302] LustreError: 170471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7201.528557] Lustre: server umount lustre-MDT0001 complete [ 7209.372342] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 20:46:38 (1787618798) [ 7215.459142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7215.786358] LustreError: 174858:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7215.802173] LustreError: 174858:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 30 previous similar messages [ 7219.715896] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 7226.997491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7231.339806] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 7238.159835] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7243.557222] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7250.011569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7256.825376] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 7266.266978] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7277.986151] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 20:47:47 (1787618867) [ 7283.178905] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7283.184358] LustreError: Skipped 2 previous similar messages [ 7283.192716] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7283.212933] Lustre: Skipped 2 previous similar messages [ 7283.220406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7283.226920] Lustre: Skipped 6 previous similar messages [ 7288.517707] Lustre: server umount lustre-OST0000 complete [ 7307.231581] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7307.480377] Lustre: server umount lustre-MDT0000 complete [ 7313.307409] LustreError: 174838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787618905 with bad export cookie 9686016983052403247 [ 7313.314057] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7313.325280] LustreError: 174838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7314.297907] Lustre: server umount lustre-MDT0001 complete [ 7328.046542] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 7332.150828] Key type lgssc unregistered [ 7332.379646] LNet: 178590:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7332.385221] LNetError: 178590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7332.414208] LNet: Removed LNI 192.168.206.128@tcp [ 7333.463531] Key type .llcrypt unregistered [ 7333.465220] Key type ._llcrypt unregistered [ 7352.054951] Key type ._llcrypt registered [ 7352.068295] Key type .llcrypt registered [ 7352.259392] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 7354.137410] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7354.182936] alg: No test for adler32 (adler32-zlib) [ 7355.493261] Lustre: Lustre: Build Version: 2.17.57_80_g20f51fc [ 7356.045774] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 7357.872191] Key type lgssc registered [ 7359.215686] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7369.153421] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7369.189994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7385.055830] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7390.176061] LustreError: 179839:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.128@tcp: failed processing log, type 4: rc = -110 [ 7395.296274] LustreError: 179839:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7395.307280] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7395.311804] LustreError: MGC192.168.206.128@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7395.331541] LustreError: 179839:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7395.340590] LustreError: 179839:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -22 [ 7395.345648] LustreError: 179839:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 7395.414111] Lustre: server umount lustre-OST0000 complete [ 7395.418657] LustreError: 179839:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7403.117136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7404.477127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7408.498570] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 7414.773873] LustreError: 180338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7414.788689] LustreError: 180338:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7416.896948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7417.306609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7421.181944] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 7428.657571] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7434.044616] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7440.232300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7440.351459] LustreError: 182155:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7440.359313] Lustre: 182155:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.206.128@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7440.451440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7446.729536] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 7454.555536] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7458.274472] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7458.288503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7458.304376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7460.325231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7460.325816] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7460.371039] Lustre: Skipped 1 previous similar message [ 7462.748578] Lustre: server umount lustre-MDT0000 complete [ 7465.452210] LustreError: 181772:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7465.484552] LustreError: 181772:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7467.153510] LustreError: 180320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787619059 with bad export cookie 3006750555060583783 [ 7467.158647] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7467.572813] Lustre: server umount lustre-MDT0001 complete [ 7483.231644] Lustre: server umount lustre-OST0000 complete [ 7494.673344] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [ 7504.507817] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 7539.663659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7540.133646] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7540.173117] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7540.336340] Lustre: lustre-MDT0000: new disk, initializing [ 7540.452874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7540.489673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7545.301568] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 7555.311678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7555.430455] Lustre: 185681:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7555.478249] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7555.481216] Lustre: Skipped 1 previous similar message [ 7555.540319] Lustre: lustre-MDT0001: new disk, initializing [ 7555.613912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7555.657761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7555.673177] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7559.568783] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 7568.314227] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7573.914325] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7579.960187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7580.295971] Lustre: lustre-OST0000: new disk, initializing [ 7580.305168] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7580.320475] Lustre: 187645:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7580.447023] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7586.840599] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 7587.404198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7587.419940] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7587.576828] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7596.371479] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7600.728289] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7601.056161] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7607.149373] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7607.403763] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7613.411175] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7613.421889] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7613.430792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7613.436885] Lustre: Skipped 3 previous similar messages [ 7615.481672] Lustre: server umount lustre-OST0000 complete [ 7622.114566] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7622.121784] LustreError: Skipped 1 previous similar message [ 7622.127051] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7622.141125] Lustre: Skipped 1 previous similar message [ 7622.163919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7628.769349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7628.773811] Lustre: Skipped 3 previous similar messages [ 7633.375481] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7633.539687] Lustre: server umount lustre-MDT0000 complete [ 7633.891976] LustreError: 185686:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7633.910569] LustreError: 185686:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7637.195899] LustreError: 185672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787619229 with bad export cookie 3006750555060584497 [ 7637.207048] LustreError: 185672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7637.207307] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7637.613419] Lustre: server umount lustre-MDT0001 complete [ 7646.489806] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 20:53:55 (1787619235) [ 7667.867729] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 7671.843728] Key type lgssc unregistered [ 7672.302660] LNet: 190600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7672.323245] LNetError: 190600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7673.384715] LNet: Removed LNI 192.168.206.128@tcp [ 7674.809742] Key type .llcrypt unregistered [ 7674.811927] Key type ._llcrypt unregistered [ 7696.372377] Key type ._llcrypt registered [ 7696.375458] Key type .llcrypt registered [ 7696.526542] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 7697.466146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7697.537054] alg: No test for adler32 (adler32-zlib) [ 7698.591810] Lustre: Lustre: Build Version: 2.17.57_80_g20f51fc [ 7698.793893] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 7700.463249] Key type lgssc registered [ 7701.545426] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7708.239586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7708.264341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7709.633209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7713.651867] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 7719.914802] LustreError: 191818:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7719.943559] LustreError: 191818:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7722.399496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7722.689382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7727.164385] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 7734.135329] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7738.655516] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7745.230980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7745.431154] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7745.434909] LustreError: 193684:0:(llog.c:1696:llog_backup()) MGC192.168.206.128@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7745.441109] Lustre: 193684:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.206.128@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7745.570755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7751.929620] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 7757.768587] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7759.840657] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7759.845868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7759.854720] Lustre: Skipped 1 previous similar message [ 7759.866607] Lustre: Skipped 2 previous similar messages [ 7764.962280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7765.694140] Lustre: server umount lustre-MDT0000 complete [ 7768.533409] LustreError: 192555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787619360 with bad export cookie 10348444638368265212 [ 7768.537285] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7768.544524] LustreError: 192555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7768.795987] Lustre: server umount lustre-MDT0001 complete [ 7781.276814] Lustre: server umount lustre-OST0000 complete [ 7789.024740] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [ 7798.155555] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 7836.244652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7836.459921] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7836.486585] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7836.561266] Lustre: lustre-MDT0000: new disk, initializing [ 7836.631653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7836.650416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7841.589233] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 7852.065553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7852.206385] Lustre: 197209:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7852.264615] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7852.269064] Lustre: Skipped 1 previous similar message [ 7852.274598] LustreError: 197989:0:(llog.c:1696:llog_backup()) MGC192.168.206.128@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7852.281428] LustreError: 197989:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7852.286876] LustreError: 197218:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7852.309553] Lustre: 197989:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.128@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7852.326587] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7852.330258] Lustre: Skipped 1 previous similar message [ 7852.411977] Lustre: lustre-MDT0001: new disk, initializing [ 7852.490424] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7852.534436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7852.543401] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7857.124377] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 7864.678562] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7868.942749] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7874.078519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7874.338224] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7874.346566] Lustre: Skipped 1 previous similar message [ 7874.353114] LustreError: 199149:0:(llog.c:1696:llog_backup()) MGC192.168.206.128@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7874.365709] LustreError: 199149:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7874.369872] Lustre: 199149:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.128@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7874.377163] Lustre: 199149:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7874.402395] Lustre: lustre-OST0000: new disk, initializing [ 7874.407235] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7874.413092] Lustre: 199173:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7874.479751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7880.662696] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 7881.798628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7881.813690] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7881.963050] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7890.142996] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7895.303754] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7895.537769] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7900.096862] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7900.340411] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7902.688966] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7902.692291] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7902.698361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7902.698368] Lustre: Skipped 2 previous similar messages [ 7902.717114] Lustre: Skipped 2 previous similar messages [ 7907.810851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7907.818024] Lustre: Skipped 1 previous similar message [ 7908.373748] Lustre: server umount lustre-OST0000 complete [ 7912.933504] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7912.942253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7912.956121] Lustre: Skipped 1 previous similar message [ 7912.967251] Lustre: Skipped 2 previous similar messages [ 7918.290383] Lustre: server umount lustre-MDT0000 complete [ 7921.630761] LustreError: 197200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787619513 with bad export cookie 10348444638368265933 [ 7921.637451] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7921.640777] LustreError: 197200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7921.949393] Lustre: server umount lustre-MDT0001 complete [ 7928.817323] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 20:58:38 (1787619518) [ 7942.559348] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 7950.623420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7950.922388] LustreError: 202297:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7950.969486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7954.530869] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 7956.448264] LustreError: 202298:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7960.551942] LustreError: 202297:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7961.573805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7966.092506] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 7968.742717] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7975.151161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7975.397219] LustreError: 203782:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7981.443430] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 7990.240450] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7990.474894] Lustre: lustre-OST0001: new disk, initializing [ 7990.477980] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7990.488391] Lustre: 204604:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7990.620019] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7990.627083] Lustre: Skipped 2 previous similar messages [ 7997.537674] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 7999.566075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7999.577721] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7999.659207] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8007.043545] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8012.757604] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8014.291662] Lustre: DEBUG MARKER: ADD OST3 [ 8017.758493] loop: module loaded [ 8017.807942] loop0: detected capacity change from 0 to 204800000 [ 8018.034188] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8018.050486] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8018.060528] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8022.388054] loop0: detected capacity change from 0 to 204800000 [ 8024.106727] Lustre: DEBUG MARKER: STOP OST3 [ 8026.662768] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8026.842987] Lustre: lustre-OST0003: new disk, initializing [ 8026.845991] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8026.850759] Lustre: 206537:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8028.292618] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8028.429554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8028.438831] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8028.481677] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8028.818268] Lustre: Failing over lustre-OST0003 [ 8028.913514] Lustre: server umount lustre-OST0003 complete [ 8030.305400] LustreError: 206680:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8032.731880] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 8033.778314] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8033.792941] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8033.811507] LustreError: 203782:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8042.362366] Lustre: DEBUG MARKER: START OST3 again [ 8049.561399] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8049.855134] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8050.985545] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8051.625230] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8051.630208] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8055.834969] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 8076.473949] Lustre: Failing over lustre-OST0003 [ 8076.556853] Lustre: server umount lustre-OST0003 complete [ 8077.280151] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8077.292113] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8077.301048] LustreError: 203782:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8077.316991] Lustre: Skipped 2 previous similar messages [ 8077.353475] LustreError: 203782:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 8095.204899] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8095.219775] LustreError: Skipped 1 previous similar message [ 8095.234217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8095.268922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8095.275920] Lustre: Skipped 2 previous similar messages [ 8101.392635] Lustre: server umount lustre-MDT0000 complete [ 8104.891358] LustreError: 202278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787619696 with bad export cookie 10348444638368266815 [ 8104.915309] LustreError: 202278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8104.919214] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8111.332805] Lustre: server umount lustre-MDT0001 complete [ 8114.898155] Lustre: server umount lustre-OST0000 complete [ 8118.937827] Lustre: server umount lustre-OST0001 complete [ 8125.183569] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [ 8131.443286] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 8169.115725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8169.384393] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8169.412558] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8169.526851] Lustre: lustre-MDT0000: new disk, initializing [ 8169.653669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8169.664760] Lustre: Skipped 2 previous similar messages [ 8169.696849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8174.828992] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 8185.126775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8185.255612] Lustre: 212069:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8185.520339] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8189.260729] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 8198.674576] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8205.158250] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8216.234504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8216.547552] Lustre: lustre-OST0000: new disk, initializing [ 8216.551386] Lustre: Skipped 1 previous similar message [ 8216.557093] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8216.562521] Lustre: Skipped 2 previous similar messages [ 8216.568510] Lustre: 214039:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8218.476824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8218.481365] Lustre: Skipped 1 previous similar message [ 8218.502620] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8218.692600] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8227.006706] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 8237.701797] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8242.483718] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8242.638685] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8246.891905] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8247.074324] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8249.320157] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8249.320505] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8249.323193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8249.323210] Lustre: Skipped 8 previous similar messages [ 8249.329918] Lustre: Skipped 6 previous similar messages [ 8249.351694] LustreError: Skipped 1 previous similar message [ 8255.553161] Lustre: server umount lustre-OST0000 complete [ 8262.119125] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8274.399287] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8274.589618] Lustre: server umount lustre-MDT0000 complete [ 8274.915236] LustreError: 212081:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8274.933667] LustreError: 212081:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 16 previous similar messages [ 8278.117279] LustreError: 212062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787619870 with bad export cookie 10348444638368269104 [ 8278.118188] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8278.124675] LustreError: 212062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8278.587157] Lustre: server umount lustre-MDT0001 complete [ 8282.523105] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 21:04:32 (1787619872) [ 8301.000158] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [ 8310.220482] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 8346.793871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8347.062712] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8347.092728] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8347.215498] Lustre: lustre-MDT0000: new disk, initializing [ 8347.293727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8347.298529] Lustre: Skipped 2 previous similar messages [ 8347.317086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8351.789728] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 8362.738252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8362.841470] Lustre: 218652:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8363.120342] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8367.327960] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 8376.565468] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8383.093129] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8391.873461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8392.356090] Lustre: 220624:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8393.710551] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8402.383765] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 8412.167644] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8417.675898] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8418.065928] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8423.483597] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8423.747123] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8429.541695] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8429.546131] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8429.557756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8429.570386] Lustre: Skipped 3 previous similar messages [ 8429.591551] Lustre: Skipped 9 previous similar messages [ 8440.287149] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8440.465298] Lustre: server umount lustre-OST0000 complete [ 8449.489444] Lustre: server umount lustre-MDT0000 complete [ 8450.017739] LustreError: 219535:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8450.045225] LustreError: 219535:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8454.650521] LustreError: 218646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787620046 with bad export cookie 10348444638368269993 [ 8454.663578] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8454.686795] LustreError: 218646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8455.681630] Lustre: server umount lustre-MDT0001 complete [ 8469.037805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8475.028382] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 8485.558539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8490.104590] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 8497.759797] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8503.833724] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8510.866632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8518.093274] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 8528.381762] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8540.129845] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8540.144906] LustreError: Skipped 2 previous similar messages [ 8540.157778] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8540.172248] Lustre: Skipped 2 previous similar messages [ 8540.179181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8540.186197] Lustre: Skipped 6 previous similar messages [ 8545.139663] Lustre: server umount lustre-OST0000 complete [ 8554.911490] Lustre: server umount lustre-MDT0000 complete [ 8559.918910] LustreError: 223577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787620151 with bad export cookie 10348444638368270875 [ 8559.923932] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8559.932164] LustreError: 223577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8560.535832] Lustre: server umount lustre-MDT0001 complete [ 8669.028658] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 8671.908741] Key type lgssc unregistered [ 8672.127762] LNet: 226513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8672.136500] LNetError: 226513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8672.159271] LNet: Removed LNI 192.168.206.128@tcp [ 8672.930364] Key type .llcrypt unregistered [ 8672.932082] Key type ._llcrypt unregistered [ 8705.887541] Key type ._llcrypt registered [ 8705.890574] Key type .llcrypt registered [ 8706.069819] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [ 8728.924307] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 8729.922886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8730.025349] alg: No test for adler32 (adler32-zlib) [ 8731.264411] Lustre: Lustre: Build Version: 2.17.57_80_g20f51fc [ 8731.680401] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 8733.447350] Key type lgssc registered [ 8734.993272] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8775.380517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8775.435528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8776.822651] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8776.866687] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8776.937921] Lustre: lustre-MDT0000: new disk, initializing [ 8777.096071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8777.132562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8782.077958] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 8792.588697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8792.686807] Lustre: 229867:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8792.756344] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8792.759543] Lustre: Skipped 1 previous similar message [ 8792.862331] Lustre: lustre-MDT0001: new disk, initializing [ 8792.926425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8792.960493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8792.965976] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8797.357925] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 8807.727716] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8815.312520] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8823.315570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8823.721247] Lustre: lustre-OST0000: new disk, initializing [ 8823.724379] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8823.735991] Lustre: 231836:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8823.842570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8830.736921] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 8831.052169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8831.060759] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8831.184307] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8843.204678] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8848.119922] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8848.306440] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8854.066723] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8854.352362] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8857.057389] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8857.062440] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8857.082863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8857.114655] Lustre: Skipped 1 previous similar message [ 8862.191895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8862.194600] Lustre: Skipped 2 previous similar messages [ 8862.947097] Lustre: server umount lustre-OST0000 complete [ 8867.298055] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8867.322286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8867.333060] Lustre: Skipped 1 previous similar message [ 8872.418276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8872.420619] Lustre: Skipped 1 previous similar message [ 8873.505278] Lustre: server umount lustre-MDT0000 complete [ 8877.987468] LustreError: 229860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787620469 with bad export cookie 1548602435619012451 [ 8877.999649] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8878.004579] LustreError: 229860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8878.658525] Lustre: server umount lustre-MDT0001 complete [ 8902.067219] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 8912.496619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8912.956228] LustreError: 234772:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8912.974985] LustreError: 234772:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8913.035664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8918.077588] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 8918.499081] LustreError: 234773:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8918.525769] LustreError: 234773:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8922.597910] LustreError: 234772:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8927.718910] LustreError: 234773:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8928.178325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8928.746346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8934.251312] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 8937.682185] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8945.731794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8946.082568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8953.023131] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 8964.212785] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8964.446766] Lustre: lustre-OST0001: new disk, initializing [ 8964.459142] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8964.468936] Lustre: 237082:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8964.604859] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8971.430446] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 8973.411840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8973.421561] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8973.508625] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8983.089684] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8993.686968] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9006.530814] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 21:16:36 (1787620596) [ 9009.632214] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9009.651789] Lustre: Skipped 2 previous similar messages [ 9009.653450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9009.681233] Lustre: Skipped 3 previous similar messages [ 9014.781817] LustreError: 234772:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9014.963500] Lustre: server umount lustre-MDT0000 complete [ 9019.652056] LustreError: 234755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787620611 with bad export cookie 1548602435619013340 [ 9019.654872] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9019.666036] LustreError: 234755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9019.876753] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9020.097343] Lustre: server umount lustre-MDT0001 complete [ 9027.146061] Lustre: server umount lustre-OST0000 complete [ 9031.765834] Lustre: server umount lustre-OST0001 complete [ 9040.365038] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [ 9051.920688] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 9095.225383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9095.510481] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9095.555891] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9095.654915] Lustre: lustre-MDT0000: new disk, initializing [ 9095.756397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9095.794797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9099.865121] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 9110.777715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9111.013857] Lustre: 241030:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9111.120993] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9111.128372] Lustre: Skipped 1 previous similar message [ 9111.295287] Lustre: lustre-MDT0001: new disk, initializing [ 9111.579193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9111.606718] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9117.004255] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 9127.518886] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9134.555835] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9140.736573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9141.011339] Lustre: lustre-OST0000: new disk, initializing [ 9141.016615] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9141.026080] Lustre: 242997:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9142.305841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9142.314828] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9142.396439] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9147.141308] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 9155.814587] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9160.546878] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9160.763296] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9165.318849] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9165.471735] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9167.840940] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9167.845590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9167.863497] Lustre: Skipped 2 previous similar messages [ 9167.866392] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9167.881899] LustreError: Skipped 1 previous similar message [ 9173.516674] Lustre: server umount lustre-OST0000 complete [ 9178.090214] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9178.098426] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9178.102149] Lustre: Skipped 1 previous similar message [ 9183.803045] Lustre: server umount lustre-MDT0000 complete [ 9187.523667] LustreError: 241022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787620779 with bad export cookie 1548602435619015475 [ 9187.528868] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9187.535356] LustreError: 241022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9187.953715] Lustre: server umount lustre-MDT0001 complete [ 9196.130943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9196.620603] LustreError: 245214:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9196.645370] LustreError: 245214:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 9196.712624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9196.718548] Lustre: Skipped 2 previous similar messages [ 9200.874554] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 9208.887066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9213.698895] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 9220.774694] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9226.436412] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9232.687549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9239.300110] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 9248.033084] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9394.158983] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9394.164467] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9394.175313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9394.183268] Lustre: Skipped 7 previous similar messages [ 9399.649874] Lustre: server umount lustre-OST0000 complete [ 9403.882295] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9403.888879] LustreError: Skipped 1 previous similar message [ 9409.777767] Lustre: server umount lustre-MDT0000 complete [ 9410.538350] LustreError: 245215:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9410.561982] LustreError: 245215:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9413.382960] LustreError: 245196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787621005 with bad export cookie 1548602435619016357 [ 9413.388579] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9413.392417] LustreError: 245196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9413.776876] Lustre: server umount lustre-MDT0001 complete [ 9517.110842] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 9519.757283] Key type lgssc unregistered [ 9520.023574] LNet: 248825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9520.029274] LNetError: 248825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9520.045466] LNet: Removed LNI 192.168.206.128@tcp [ 9520.846120] Key type .llcrypt unregistered [ 9520.851481] Key type ._llcrypt unregistered [ 9544.600699] Key type ._llcrypt registered [ 9544.604455] Key type .llcrypt registered [ 9544.708682] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [ 9560.197316] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 9561.643108] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9561.697666] alg: No test for adler32 (adler32-zlib) [ 9562.825103] Lustre: Lustre: Build Version: 2.17.57_80_g20f51fc [ 9563.024178] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 9564.680296] Key type lgssc registered [ 9565.897695] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9613.562975] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 9623.645930] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9623.672600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9625.017028] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9625.082990] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9625.202205] Lustre: lustre-MDT0000: new disk, initializing [ 9625.311143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9625.348565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9629.928718] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 9640.170927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9640.292796] Lustre: 252894:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9640.340750] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9640.344208] Lustre: Skipped 1 previous similar message [ 9640.405620] Lustre: lustre-MDT0001: new disk, initializing [ 9640.479025] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9640.519464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9640.525522] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9644.891569] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 9649.606959] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9656.265592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9656.504971] Lustre: lustre-OST0000: new disk, initializing [ 9656.508325] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9656.513443] Lustre: 254526:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9656.591874] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9660.584203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9660.594499] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9660.698072] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9662.257693] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 9672.687216] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9672.866104] Lustre: lustre-OST0001: new disk, initializing [ 9672.869628] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9672.876360] Lustre: 255396:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9672.979840] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9679.865399] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 9681.010173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9681.022715] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9681.156552] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9690.459985] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9699.413222] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9711.979226] Lustre: server umount lustre-MDT0000 complete [ 9712.101093] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9712.102631] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9712.118120] Lustre: Skipped 1 previous similar message [ 9715.013136] LustreError: 252885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787621306 with bad export cookie 9161353250871869619 [ 9715.021853] LustreError: 252885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9715.026254] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9715.399965] Lustre: server umount lustre-MDT0001 complete [ 9728.762261] Lustre: server umount lustre-OST0000 complete [ 9743.276513] Lustre: server umount lustre-OST0001 complete [ 9757.024318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9766.010875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9773.312840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9782.766065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9806.386210] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 9816.310616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9816.328283] Lustre: lustre-MDT0000: reset Object Index mappings [ 9816.702230] LustreError: 260406:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9816.717500] LustreError: 260406:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9816.801082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9821.343964] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 9822.189981] LustreError: 260407:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9826.280172] LustreError: 260406:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9830.623916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9830.651760] Lustre: lustre-MDT0001: reset Object Index mappings [ 9831.096128] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9835.798718] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 9839.164402] Lustre: 261549:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9847.081502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9847.577961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9855.389129] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 9855.792240] LustreError: 261903:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9861.100668] LustreError: 261902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9862.138470] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9864.272901] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9869.840449] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9870.890162] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 9878.213493] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9881.605556] Lustre: 263419:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9887.201332] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9887.212923] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9887.228904] Lustre: Skipped 2 previous similar messages [ 9887.233947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9889.048466] Lustre: server umount lustre-MDT0000 complete [ 9890.276544] LustreError: 261139:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9890.283074] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9890.312563] LustreError: 261139:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9890.314595] Lustre: Skipped 2 previous similar messages [ 9894.216797] LustreError: 263421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787621486 with bad export cookie 9161353250871876409 [ 9894.223704] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9894.228985] LustreError: 263421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9895.079244] Lustre: server umount lustre-MDT0001 complete [ 9911.711421] Lustre: 250403:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787621487/real 1787621487] req@ffff9fd17795a680 x1874456639302528/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787621503 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9911.763428] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9911.878143] Lustre: server umount lustre-OST0000 complete [ 9915.983509] Lustre: server umount lustre-OST0001 complete [ 9935.743783] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 21:32:05 (1787621525) [ 9952.451124] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [ 9964.577420] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [10008.460507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10008.680742] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10008.706096] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10008.789956] Lustre: lustre-MDT0000: new disk, initializing [10008.872470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10008.879578] Lustre: Skipped 1 previous similar message [10008.894987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10014.245365] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10025.454859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10025.618984] Lustre: 267672:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10025.718206] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10025.727914] Lustre: Skipped 1 previous similar message [10025.851537] Lustre: lustre-MDT0001: new disk, initializing [10026.049100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10026.067103] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10031.289217] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10041.408877] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10047.754240] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10055.644310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10055.902436] Lustre: lustre-OST0000: new disk, initializing [10055.907847] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10055.917367] Lustre: 269642:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10057.534460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10057.540238] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10057.663991] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10062.740894] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10073.892714] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10079.761704] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10079.935738] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10084.634330] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10084.843557] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10088.419606] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10088.431078] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10088.437857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10088.457521] Lustre: Skipped 1 previous similar message [10093.542541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10093.551122] Lustre: Skipped 2 previous similar messages [10098.664403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10098.674520] Lustre: Skipped 1 previous similar message [10101.216194] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10101.529462] Lustre: server umount lustre-OST0000 complete [10107.873509] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10107.886358] LustreError: Skipped 1 previous similar message [10107.888706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10107.919103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10112.314752] Lustre: server umount lustre-MDT0000 complete [10114.023609] LustreError: 267684:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10114.040090] LustreError: 267684:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10116.882358] LustreError: 267666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787621708 with bad export cookie 9161353250871878502 [10116.891537] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10116.900827] LustreError: 267666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10117.381288] Lustre: server umount lustre-MDT0001 complete [10140.785969] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [10151.117025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10151.490226] LustreError: 272581:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10151.544537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10151.557356] Lustre: Skipped 2 previous similar messages [10156.267894] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10165.778372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10171.157176] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10174.303730] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10181.558811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10188.637316] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10197.488929] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10197.639097] Lustre: lustre-OST0001: new disk, initializing [10197.644089] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10197.653705] Lustre: 274890:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10199.575577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10199.604854] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10199.750761] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10204.409687] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10213.767653] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10217.697691] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10223.921197] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10224.095668] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10232.800993] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10232.810176] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10232.822873] Lustre: Skipped 1 previous similar message [10232.829596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10232.831849] Lustre: Skipped 2 previous similar messages [10237.727925] Lustre: server umount lustre-MDT0000 complete [10241.000052] LustreError: 273309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10241.042688] LustreError: 273309:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10242.097969] LustreError: 272562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787621833 with bad export cookie 9161353250871879384 [10242.102285] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10242.107826] LustreError: 272562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10242.530663] Lustre: server umount lustre-MDT0001 complete [10257.598276] Lustre: server umount lustre-OST0000 complete [10262.496261] Lustre: 250404:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787621838/real 1787621838] req@ffff9fd145784e00 x1874456639415936/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787621854 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10262.958692] Lustre: server umount lustre-OST0001 complete [10282.171122] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [10293.598444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10294.166417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10294.174561] Lustre: Skipped 3 previous similar messages [10299.397755] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10308.639606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10314.577194] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10317.609615] Lustre: 279091:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10325.066142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10332.328517] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10338.931302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10341.820373] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10347.549190] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10349.023655] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10357.263059] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10361.587037] Lustre: 280968:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10367.091803] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10367.100780] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10378.292211] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 21:39:27 (1787621967) [10396.133874] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10396.157450] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10396.174244] Lustre: Skipped 4 previous similar messages [10396.183950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10396.187225] Lustre: Skipped 4 previous similar messages [10400.029070] Lustre: server umount lustre-MDT0000 complete [10403.810868] LustreError: 281435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10403.852486] LustreError: 281435:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [10404.393991] LustreError: 277927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787621996 with bad export cookie 9161353250871885964 [10404.397470] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10404.411161] LustreError: 277927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10410.891357] Lustre: server umount lustre-MDT0001 complete [10421.566798] Lustre: server umount lustre-OST0000 complete [10425.990567] Lustre: server umount lustre-OST0001 complete [10433.059429] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [10443.412801] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [10487.769118] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [10496.781269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10497.000668] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10497.040191] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10497.132617] Lustre: lustre-MDT0000: new disk, initializing [10497.255739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10501.165245] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10512.331480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10512.522788] Lustre: 285210:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10512.844018] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10517.697539] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10532.475209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10532.759807] Lustre: 286841:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10534.572873] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10540.665657] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10551.349769] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10551.591103] Lustre: 287712:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10551.693821] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10551.700895] Lustre: Skipped 6 previous similar messages [10553.358670] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10553.364802] Lustre: Skipped 1 previous similar message [10553.495572] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10558.508704] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10568.728782] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10573.402345] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10573.410833] Lustre: Skipped 1 previous similar message [10589.429705] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 21:42:58 (1787622178) [10594.190611] Lustre: server umount lustre-MDT0000 complete [10594.296515] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10594.316496] Lustre: Skipped 5 previous similar messages [10599.077479] LustreError: 285202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787622190 with bad export cookie 9161353250871893622 [10599.087936] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10599.094072] LustreError: 285202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10599.666780] Lustre: server umount lustre-MDT0001 complete [10606.044637] Lustre: server umount lustre-OST0000 complete [10610.685378] Lustre: server umount lustre-OST0001 complete [10620.306238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10625.484390] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10635.401723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10640.719678] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10650.664931] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10657.742585] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10665.732265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10674.519703] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10680.659688] LustreError: 291793:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10680.708176] LustreError: 291793:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 16 previous similar messages [10683.392217] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10717.150194] loop1: detected capacity change from 0 to 204800000 [10717.498387] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10717.517256] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10717.571363] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10723.269181] loop1: detected capacity change from 0 to 204800000 [10730.649211] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10730.814104] LustreError: 289947:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs28183: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10730.823653] LustreError: 289947:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs28183 [10730.828044] LustreError: 289947:0:(mgs_handler.c:517:mgs_target_reg()) fs28183-MDT0001: Error checking target index: rc = -13 [10730.834626] LustreError: fs28183-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10730.839438] LustreError: 293704:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [10730.844483] LustreError: 293704:0:(tgt_mount.c:2132:server_put_super()) no obd fs28183-MDT0001 [10730.848863] LustreError: 293704:0:(tgt_mount.c:116:server_deregister_mount()) fs28183-MDT0001 not registered [10730.880471] Lustre: server umount fs28183-MDT0001 complete [10730.883743] LustreError: 293704:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10742.169375] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10742.349924] Lustre: Setting parameter fs28183-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs28183-MDT0001 [10742.470258] Lustre: srv-fs28183-MDT0001: No data found on store. Initialize space. [10742.480412] Lustre: Skipped 4 previous similar messages [10742.675338] Lustre: fs28183-MDT0001: new disk, initializing [10742.685433] Lustre: Skipped 3 previous similar messages [10742.940196] Lustre: srv-fs28183-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10742.955924] Lustre: 294171:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs28183-MDT0001-mdtlov: can't create new llog: rc = -115 [10742.967923] LustreError: 294171:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs28183-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10748.362926] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10758.012473] Lustre: server umount fs28183-MDT0001 complete [10762.740411] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10773.838877] loop1: detected capacity change from 0 to 204800000 [10781.263857] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10781.807812] Lustre: 295739:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs28183-MDT0001-mdtlov: can't create new llog: rc = -115 [10781.823700] LustreError: 295739:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs28183-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10787.287568] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10792.680239] Lustre: server umount fs28183-MDT0001 complete [10803.687585] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10803.701484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10803.706843] Lustre: Skipped 6 previous similar messages [10803.729306] Lustre: Skipped 10 previous similar messages [10809.028049] Lustre: server umount lustre-MDT0000 complete [10818.831219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10818.903772] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10824.088658] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10824.688250] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10832.197366] loop1: detected capacity change from 0 to 204800000 [10838.689964] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10839.275823] Lustre: 298184:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs28183-MDT0001-mdtlov: can't create new llog: rc = -115 [10839.290251] LustreError: 298184:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs28183-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10844.779236] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10850.449574] Lustre: server umount fs28183-MDT0001 complete [10880.481287] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10880.485421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10880.494719] LustreError: Skipped 2 previous similar messages [10880.512150] Lustre: Skipped 8 previous similar messages [10881.886351] Lustre: server umount lustre-OST0000 complete [10892.758058] Lustre: server umount lustre-MDT0000 complete [10896.854945] LustreError: 289941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787622488 with bad export cookie 9161353250871896303 [10896.865676] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10896.868323] LustreError: 289941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10897.235836] Lustre: server umount lustre-MDT0001 complete [10908.586673] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [10911.569745] Key type lgssc unregistered [10911.857672] LNet: 300534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10911.861474] LNetError: 300534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10911.871884] LNet: Removed LNI 192.168.206.128@tcp [10912.785170] Key type .llcrypt unregistered [10912.787338] Key type ._llcrypt unregistered [10932.334695] Key type ._llcrypt registered [10932.337882] Key type .llcrypt registered [10932.461654] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [10933.649636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10933.667587] alg: No test for adler32 (adler32-zlib) [10934.703654] Lustre: Lustre: Build Version: 2.17.57_80_g20f51fc [10934.883385] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [10936.495174] Key type lgssc registered [10937.837915] Lustre: Echo OBD driver; http://www.lustre.org/ [10946.886293] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10946.901173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10948.499501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10953.648518] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10958.822190] LustreError: 301827:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10958.850469] LustreError: 301827:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10963.384026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10963.789899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10969.497845] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10978.038500] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10985.971394] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10993.918716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10994.624496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11001.953474] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [11008.942780] LustreError: 303658:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11010.181331] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11014.121869] LustreError: 303659:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11018.228650] LustreError: 304280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11018.260190] LustreError: 304280:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11023.335897] LustreError: 303660:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11023.354984] LustreError: 303660:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11032.109571] LustreError: 303659:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11032.135516] LustreError: 303659:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11039.279728] loop1: detected capacity change from 0 to 204800000 [11039.425990] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11039.435557] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11039.449103] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11044.872489] loop1: detected capacity change from 0 to 204800000 [11048.945952] LustreError: 303660:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11048.965179] LustreError: 303660:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [11050.916412] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11051.108395] Lustre: fs4121-OST0000: new disk, initializing [11051.112113] Lustre: srv-fs4121-OST0000: No data found on store. Initialize space. [11051.118416] Lustre: 305586:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11051.215934] Lustre: fs4121-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11057.968981] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [11065.956353] Lustre: server umount fs4121-OST0000 complete [11080.007405] loop1: detected capacity change from 0 to 204800000 [11083.338442] LustreError: 305489:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11083.373133] LustreError: 305489:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [11086.558490] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11086.928864] Lustre: fs4121-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11089.906345] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11094.265868] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [11095.020354] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11103.877689] loop2: detected capacity change from 0 to 204800000 [11104.078751] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11104.091454] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11104.102819] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11110.692995] loop2: detected capacity change from 0 to 204800000 [11117.444522] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11117.516671] LustreError: fs4121-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11117.549255] LustreError: fs4121-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11117.563182] LustreError: 308624:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [11117.577346] LustreError: 308624:0:(tgt_mount.c:2132:server_put_super()) no obd fs4121-OST0001 [11117.591651] LustreError: 308624:0:(tgt_mount.c:116:server_deregister_mount()) fs4121-OST0001 not registered [11117.669255] Lustre: server umount fs4121-OST0001 complete [11117.675094] LustreError: 308624:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11130.600103] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11130.992914] Lustre: fs4121-OST0001: new disk, initializing [11131.000281] Lustre: srv-fs4121-OST0001: No data found on store. Initialize space. [11131.011687] Lustre: 309176:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11131.099673] Lustre: fs4121-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11139.783764] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [11149.868675] LustreError: 305489:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11149.887909] LustreError: 305489:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [11150.033964] Lustre: server umount fs4121-OST0000 complete [11159.476285] Lustre: server umount fs4121-OST0001 complete [11172.257427] loop1: detected capacity change from 0 to 204800000 [11179.917749] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11180.277626] Lustre: fs4121-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11187.868296] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [11196.528572] loop2: detected capacity change from 0 to 204800000 [11203.141452] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11210.852853] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [11225.595941] Lustre: server umount fs4121-OST0000 complete [11237.689456] Lustre: server umount fs4121-OST0001 complete [11243.506970] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11248.625224] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11253.730148] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11253.752890] Lustre: Skipped 1 previous similar message [11253.756249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11257.506968] Lustre: server umount lustre-OST0000 complete [11263.973734] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11263.976347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11263.982423] Lustre: Skipped 1 previous similar message [11263.990607] Lustre: Skipped 2 previous similar messages [11268.854940] Lustre: server umount lustre-MDT0000 complete [11272.766640] LustreError: 301806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787622864 with bad export cookie 7127066246105430406 [11272.769649] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11272.780302] LustreError: 301806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11273.182169] Lustre: server umount lustre-MDT0001 complete [11286.848382] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [11290.149455] Key type lgssc unregistered [11290.478506] LNet: 315106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11290.484213] LNetError: 315106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11290.509239] LNet: Removed LNI 192.168.206.128@tcp [11291.879164] Key type .llcrypt unregistered [11291.882597] Key type ._llcrypt unregistered [11327.855385] Key type ._llcrypt registered [11327.860623] Key type .llcrypt registered [11328.007362] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [11344.570247] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [11345.685795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11345.812791] alg: No test for adler32 (adler32-zlib) [11346.845975] Lustre: Lustre: Build Version: 2.17.57_80_g20f51fc [11347.185041] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [11348.895174] Key type lgssc registered [11349.948819] Lustre: Echo OBD driver; http://www.lustre.org/ [11386.670852] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11386.693284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11387.994364] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11388.015399] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11388.148211] Lustre: lustre-MDT0000: new disk, initializing [11388.260387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11388.278692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11392.317366] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [11402.953569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11403.088212] Lustre: 318966:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11403.140644] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11403.144817] Lustre: Skipped 1 previous similar message [11403.229224] Lustre: lustre-MDT0001: new disk, initializing [11403.338231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11403.381421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11403.389599] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11407.970095] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [11416.300238] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11421.134237] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11427.725018] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11428.077801] Lustre: lustre-OST0000: new disk, initializing [11428.086531] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11428.107045] Lustre: 320934:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11428.291751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11434.586381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11434.600461] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11434.802401] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11435.657960] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [11448.148674] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11450.296893] Lustre: Mounted lustre-client - version 2.17.57_80_g20f51fc [11451.370156] Lustre: Unmounted lustre-client [11465.701661] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11465.713883] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11465.725936] LustreError: Skipped 1 previous similar message [11465.736862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11465.795447] Lustre: Skipped 1 previous similar message [11465.813645] Lustre: Skipped 1 previous similar message [11468.422877] Lustre: server umount lustre-OST0000 complete [11474.915857] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11474.934704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11474.962484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11475.946172] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11475.970067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11475.974987] Lustre: Skipped 1 previous similar message [11479.597348] Lustre: server umount lustre-MDT0000 complete [11483.427624] LustreError: 318958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787623075 with bad export cookie 14571411541612651491 [11483.432155] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11483.453721] LustreError: 318958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11483.683889] Lustre: server umount lustre-MDT0001 complete [11493.580353] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [11497.018667] Key type lgssc unregistered [11503.968730] LNet: 1 local NIs in recovery (showing 1): 192.168.206.128@tcp [11504.481072] LNet: 323097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11504.504399] LNetError: 323097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11581.353522] LNet: Removed LNI 192.168.206.128@tcp [11582.913696] Key type .llcrypt unregistered [11582.919309] Key type ._llcrypt unregistered [11608.308247] Key type ._llcrypt registered [11608.311437] Key type .llcrypt registered [11608.439427] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [11623.338405] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [11624.972298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11625.010832] alg: No test for adler32 (adler32-zlib) [11626.191518] Lustre: Lustre: Build Version: 2.17.57_80_g20f51fc [11626.538210] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [11628.279184] Key type lgssc registered [11629.695598] Lustre: Echo OBD driver; http://www.lustre.org/ [11663.975286] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 22:00:53 (1787623253) [11673.012465] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11673.052675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11674.385832] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11674.433199] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11674.586934] Lustre: lustre-MDT0000: new disk, initializing [11674.663143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11674.684436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11680.894470] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [11694.619642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11694.749701] Lustre: 326333:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11694.803304] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11694.807594] Lustre: Skipped 1 previous similar message [11694.873476] Lustre: lustre-MDT0001: new disk, initializing [11694.973860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11695.010204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11695.025271] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11700.277204] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [11711.504527] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11717.768671] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11726.313576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11726.559991] Lustre: lustre-OST0000: new disk, initializing [11726.565269] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11726.571458] Lustre: 328299:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11726.708552] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11733.077964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11733.103631] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11733.244226] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11737.884607] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [11751.770849] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11774.438969] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11774.444948] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11774.455176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11774.461927] Lustre: Skipped 1 previous similar message [11774.471778] Lustre: Skipped 1 previous similar message [11778.800920] Lustre: server umount lustre-OST0000 complete [11784.672591] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11784.689415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11784.701179] Lustre: Skipped 1 previous similar message [11789.793888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11789.801889] Lustre: Skipped 1 previous similar message [11790.956571] Lustre: server umount lustre-MDT0000 complete [11795.392151] LustreError: 326322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787623387 with bad export cookie 11701395016567769685 [11795.406909] LustreError: 326322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11795.423174] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11795.835726] Lustre: server umount lustre-MDT0001 complete [11810.460131] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [11813.544790] Key type lgssc unregistered [11813.975564] LNet: 330216:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11813.992079] LNetError: 330216:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11815.014171] LNet: Removed LNI 192.168.206.128@tcp [11815.981586] Key type .llcrypt unregistered [11815.984395] Key type ._llcrypt unregistered [11839.822468] Key type ._llcrypt registered [11839.824568] Key type .llcrypt registered [11839.920627] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [11841.643267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11841.778181] alg: No test for adler32 (adler32-zlib) [11842.892402] Lustre: Lustre: Build Version: 2.17.57_80_g20f51fc [11843.081514] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [11844.831206] Key type lgssc registered [11845.879680] Lustre: Echo OBD driver; http://www.lustre.org/ [11854.605562] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11854.621612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11856.440708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11863.006529] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [11866.597417] LustreError: 331490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11866.612406] LustreError: 331490:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11871.726714] LustreError: 331491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11873.401278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11873.902402] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11878.873632] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [11888.867958] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11899.646284] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11911.580251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11912.405306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11921.991312] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [11931.977561] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11942.916683] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11944.878722] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11950.156623] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.28@tcp (stopping) [11950.572430] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11950.577291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11952.619291] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11952.638938] Lustre: Skipped 1 previous similar message [11952.646184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11952.651750] Lustre: Skipped 3 previous similar messages [11954.705893] Lustre: server umount lustre-MDT0000 complete [11955.239565] LustreError: 331486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11957.738237] LustreError: 332888:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11957.761948] LustreError: 332888:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11958.714108] LustreError: 331473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787623550 with bad export cookie 18157206903970962625 [11958.716253] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11958.723666] LustreError: 331473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11960.410669] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.28@tcp (stopping) [11962.849566] LustreError: 331485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11962.849986] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11962.873332] LustreError: 331485:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11962.911625] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11968.995711] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11968.999272] Lustre: Skipped 2 previous similar messages [11971.484223] LustreError: 331485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11971.518272] LustreError: 331485:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11973.087955] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11973.370704] Lustre: server umount lustre-MDT0001 complete [11977.677799] Lustre: server umount lustre-OST0000 complete [11987.559433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11992.400709] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [11995.107076] Lustre: 335676:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.128@tcp [11997.368251] Lustre: server umount MGS complete [12005.683148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12006.074580] LustreError: 336248:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12006.169876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12010.524154] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [12018.657160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12019.029485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12024.016832] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [12032.960746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12033.303883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12040.201570] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [12048.884501] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12050.682831] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12056.688211] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12058.516313] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12064.224785] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12064.234707] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12064.255576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12064.258991] Lustre: Skipped 2 previous similar messages [12065.879632] Lustre: server umount lustre-OST0000 complete [12070.368733] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12070.382611] LustreError: Skipped 1 previous similar message [12070.392029] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12070.403442] Lustre: Skipped 1 previous similar message [12076.421210] Lustre: server umount lustre-MDT0000 complete [12076.513598] LustreError: 336248:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12076.531453] LustreError: 336248:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12081.174723] LustreError: 336229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787623673 with bad export cookie 18157206903970963465 [12081.188105] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12081.555623] Lustre: server umount lustre-MDT0001 complete [12100.721188] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [12105.488702] Key type lgssc unregistered [12106.082594] LNet: 339700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12106.091184] LNetError: 339700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12106.130168] LNet: Removed LNI 192.168.206.128@tcp [12107.518180] Key type .llcrypt unregistered [12107.521044] Key type ._llcrypt unregistered [12131.265148] Key type ._llcrypt registered [12131.267383] Key type .llcrypt registered [12131.403668] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [12154.375717] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [12155.777762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12155.936465] alg: No test for adler32 (adler32-zlib) [12157.515279] Lustre: Lustre: Build Version: 2.17.57_80_g20f51fc [12157.955721] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [12159.791166] Key type lgssc registered [12161.106310] Lustre: Echo OBD driver; http://www.lustre.org/ [12204.813193] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12204.839401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12206.212436] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12206.235516] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12206.300531] Lustre: lustre-MDT0000: new disk, initializing [12206.357850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12206.372536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12211.210358] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [12222.349494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12222.430037] Lustre: 342863:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12222.489500] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12222.494347] Lustre: Skipped 1 previous similar message [12222.566995] Lustre: lustre-MDT0001: new disk, initializing [12222.648211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12222.683528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12222.695952] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12228.232359] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [12238.651712] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12244.682541] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12251.676672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12251.948785] Lustre: lustre-OST0000: new disk, initializing [12251.957905] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12251.964399] Lustre: 344830:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12252.048137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12258.846062] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [12260.409748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12260.424376] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12260.524469] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12269.538887] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12276.878937] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12277.481378] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12285.731311] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12286.161688] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12291.552889] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12291.558341] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12291.594522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12291.611822] Lustre: Skipped 1 previous similar message [12296.676708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12296.682678] Lustre: Skipped 1 previous similar message [12301.803089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12301.814108] Lustre: Skipped 1 previous similar message [12303.327764] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12303.569035] Lustre: server umount lustre-OST0000 complete [12309.993853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12310.010806] LustreError: Skipped 1 previous similar message [12310.018484] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12310.034938] Lustre: Skipped 1 previous similar message [12310.047532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12312.037097] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12317.162308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12317.185754] Lustre: Skipped 3 previous similar messages [12323.295147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12323.737976] Lustre: server umount lustre-MDT0000 complete [12328.857425] LustreError: 342854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787623920 with bad export cookie 918898269955241439 [12328.892426] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12329.347562] Lustre: server umount lustre-MDT0001 complete [12336.341454] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 22:12:06 (1787623926) [12348.783903] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [12361.078467] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [12433.793363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12434.478181] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12434.561921] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12434.882442] Lustre: lustre-MDT0000: new disk, initializing [12435.034334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12435.092937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12444.996044] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [12461.266221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12461.409330] Lustre: 349598:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12461.469244] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12461.473334] Lustre: Skipped 1 previous similar message [12461.544716] Lustre: lustre-MDT0001: new disk, initializing [12461.644802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12461.674749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12461.693462] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12468.275423] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [12478.491913] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12484.232978] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12490.940092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12491.166435] Lustre: lustre-OST0000: new disk, initializing [12491.171827] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12491.176828] Lustre: 351570:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12491.275460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12496.980682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12496.996291] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12497.153142] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12497.612560] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [12508.394130] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12548.582329] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12548.587309] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12548.604780] LustreError: Skipped 1 previous similar message [12548.610188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12548.610194] Lustre: Skipped 2 previous similar messages [12559.327132] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12559.632678] Lustre: server umount lustre-OST0000 complete [12569.059155] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12569.064323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12569.066123] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12569.073285] Lustre: Skipped 2 previous similar messages [12569.078206] Lustre: Skipped 5 previous similar messages [12570.933791] Lustre: server umount lustre-MDT0000 complete [12574.183713] LustreError: 349611:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12574.204615] LustreError: 349611:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12574.916699] LustreError: 349592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787624166 with bad export cookie 918898269955242321 [12574.924038] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12574.955271] LustreError: 349592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [12575.575084] Lustre: server umount lustre-MDT0001 complete [12588.210161] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [12592.132861] Key type lgssc unregistered [12592.442145] LNet: 353749:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12592.458588] LNetError: 353749:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12592.484783] LNet: Removed LNI 192.168.206.128@tcp [12593.139290] Key type .llcrypt unregistered [12593.142776] Key type ._llcrypt unregistered [12691.810148] Key type ._llcrypt registered [12691.812649] Key type .llcrypt registered [12691.915267] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [12692.268651] Key type .llcrypt unregistered [12692.273521] Key type ._llcrypt unregistered [12779.533505] Key type ._llcrypt registered [12779.536579] Key type .llcrypt registered [12779.689469] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [12780.152389] Key type .llcrypt unregistered [12780.154742] Key type ._llcrypt unregistered [12817.936552] Key type ._llcrypt registered [12817.938239] Key type .llcrypt registered [12818.024806] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [12818.306825] Key type .llcrypt unregistered [12818.308412] Key type ._llcrypt unregistered [12853.916273] Key type ._llcrypt registered [12853.918718] Key type .llcrypt registered [12854.234675] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [12854.844347] Key type .llcrypt unregistered [12854.849474] Key type ._llcrypt unregistered [12956.486296] Key type ._llcrypt registered [12956.488579] Key type .llcrypt registered [12956.624862] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [12956.985350] Key type .llcrypt unregistered [12956.988651] Key type ._llcrypt unregistered [12987.354270] Key type ._llcrypt registered [12987.360174] Key type .llcrypt registered [12987.567617] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [12987.940719] Key type .llcrypt unregistered [12987.947448] Key type ._llcrypt unregistered [13021.537400] Key type ._llcrypt registered [13021.541372] Key type .llcrypt registered [13021.687726] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [13022.011408] Key type .llcrypt unregistered [13022.018855] Key type ._llcrypt unregistered [13051.997820] Key type ._llcrypt registered [13052.002091] Key type .llcrypt registered [13052.122901] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [13052.421252] Key type .llcrypt unregistered [13052.423670] Key type ._llcrypt unregistered [13083.479769] Key type ._llcrypt registered [13083.482181] Key type .llcrypt registered [13083.570415] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [13083.914299] Key type .llcrypt unregistered [13083.918569] Key type ._llcrypt unregistered [13120.767590] Key type ._llcrypt registered [13120.770288] Key type .llcrypt registered [13120.903230] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [13121.420182] Key type .llcrypt unregistered [13121.422887] Key type ._llcrypt unregistered [13184.711188] Key type ._llcrypt registered [13184.715227] Key type .llcrypt registered [13184.940491] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [13185.440359] Key type .llcrypt unregistered [13185.442633] Key type ._llcrypt unregistered [13214.686657] Key type ._llcrypt registered [13214.690882] Key type .llcrypt registered [13214.843178] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [13215.177629] Key type .llcrypt unregistered [13215.181134] Key type ._llcrypt unregistered [13245.052548] Key type ._llcrypt registered [13245.055087] Key type .llcrypt registered [13245.138993] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [13245.412776] Key type .llcrypt unregistered [13245.416266] Key type ._llcrypt unregistered [13295.316423] Key type ._llcrypt registered [13295.338120] Key type .llcrypt registered [13295.455266] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [13312.885780] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [13313.937565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13313.975277] alg: No test for adler32 (adler32-zlib) [13315.098233] Lustre: Lustre: Build Version: 2.17.57_80_g20f51fc [13315.347469] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [13317.049166] Key type lgssc registered [13318.311377] Lustre: Echo OBD driver; http://www.lustre.org/ [13352.860542] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13352.889285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13354.043647] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13354.070951] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13354.148433] Lustre: tst250fs-MDT0000: new disk, initializing [13354.238936] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13354.252362] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13358.205996] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [13369.596288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13369.745487] Lustre: 374141:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13369.843633] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13369.850850] Lustre: Skipped 1 previous similar message [13370.036606] Lustre: tst250fs-MDT0001: new disk, initializing [13370.189521] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13370.271709] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13370.313622] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13374.956950] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [13380.233897] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13386.709934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13387.031997] Lustre: tst250fs-OST0000: new disk, initializing [13387.037820] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13387.055480] Lustre: 375771:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13387.140259] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13388.357519] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13388.374919] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13388.526216] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13392.679380] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [13402.233827] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13402.425903] Lustre: tst250fs-OST0001: new disk, initializing [13402.431977] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13402.444551] Lustre: 376646:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13402.558241] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13407.977900] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [13411.455165] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13411.475570] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13411.695963] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13413.869646] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13415.887361] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [13415.891962] Lustre: Skipped 1 previous similar message [13442.016918] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13442.023593] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13442.038721] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13442.547079] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13442.573142] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13442.579784] Lustre: Skipped 2 previous similar messages [13445.561141] Lustre: server umount tst250fs-MDT0000 complete [13449.524457] LustreError: 374944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787625041 with bad export cookie 16247013169988198138 [13449.530753] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13449.538782] LustreError: 374944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13449.859621] Lustre: server umount tst250fs-MDT0001 complete [13464.106845] Lustre: server umount tst250fs-OST0000 complete [13468.127137] Lustre: 372363:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787625044/real 1787625044] req@ffff9fd17cb50000 x1874460574551808/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787625060 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13468.200358] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13468.245326] Lustre: Skipped 2 previous similar messages [13468.656281] Lustre: server umount tst250fs-OST0001 complete [13491.404330] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [13500.499348] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [13534.028436] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 22:32:03 (1787625123) [13535.574136] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13552.627202] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [13564.345622] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [13596.178793] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13318 sec ========= 22:33:05 (1787625185) [13597.592270] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 22:33:07 (1787625187) === [13600.653357] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 22:33:10 (1787625190) === [13635.324539] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [13637.958987] Key type lgssc unregistered [13638.228829] LNet: 385791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13638.243091] LNetError: 385791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13638.277125] LNet: Removed LNI 192.168.206.128@tcp [13639.077263] Key type .llcrypt unregistered [13639.079228] Key type ._llcrypt unregistered