[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 508968872 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001028] APIC: Switch to symmetric I/O mode setup [ 0.002346] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.006947] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007017] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008010] pid_max: default: 32768 minimum: 301 [ 0.009116] LSM: Security Framework initializing [ 0.010048] Yama: becoming mindful. [ 0.011037] SELinux: Initializing. [ 0.013073] *** VALIDATE selinux *** [ 0.021336] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025785] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027136] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028102] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029103] *** VALIDATE tmpfs *** [ 0.030433] *** VALIDATE proc *** [ 0.032217] *** VALIDATE cgroup *** [ 0.033010] *** VALIDATE cgroup2 *** [ 0.035154] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036139] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038027] Spectre V2 : User space: Vulnerable [ 0.040005] Speculative Store Bypass: Vulnerable [ 0.042694] debug: unmapping init [mem 0xffffffff96e59000-0xffffffff96e60fff] [ 0.044138] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045634] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046022] ... version: 2 [ 0.047013] ... bit width: 48 [ 0.048013] ... generic registers: 4 [ 0.049015] ... value mask: 0000ffffffffffff [ 0.050014] ... max period: 00007fffffffffff [ 0.051015] ... fixed-purpose events: 3 [ 0.052011] ... event mask: 000000070000000f [ 0.053307] rcu: Hierarchical SRCU implementation. [ 0.055354] smp: Bringing up secondary CPUs ... [ 0.056553] x86: Booting SMP configuration: [ 0.057024] .... node #0, CPUs: #1 #2 #3 [ 0.061156] smp: Brought up 1 node, 4 CPUs [ 0.063015] smpboot: Max logical packages: 1 [ 0.064013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.136247] node 0 deferred pages initialised in 69ms [ 0.139179] devtmpfs: initialized [ 0.140211] x86/mm: Memory block size: 128MB [ 0.142197] gcov: version magic: 0x41383552 [ 0.146099] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.147131] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.151373] pinctrl core: initialized pinctrl subsystem [ 0.153259] [ 0.153893] ************************************************************* [ 0.157017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159014] ** ** [ 0.161015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.164018] ** ** [ 0.166013] ** This means that this kernel is built to expose internal ** [ 0.169018] ** IOMMU data structures, which may compromise security on ** [ 0.172027] ** your system. ** [ 0.174015] ** ** [ 0.177016] ** If you see this message and you are not debugging the ** [ 0.179014] ** kernel, report this immediately to your vendor! ** [ 0.182019] ** ** [ 0.185016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.187014] ************************************************************* [ 0.190847] NET: Registered protocol family 16 [ 0.192547] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.195080] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.198080] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.202060] cpuidle: using governor menu [ 0.203744] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.206072] PCI: Using configuration type 1 for base access [ 0.208144] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.221055] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.222018] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.224091] cryptd: max_cpu_qlen set to 1000 [ 0.227287] ACPI: Added _OSI(Module Device) [ 0.229018] ACPI: Added _OSI(Processor Device) [ 0.230056] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.232015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.238255] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.246356] ACPI: Interpreter enabled [ 0.248058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.250013] ACPI: Using IOAPIC for interrupt routing [ 0.252185] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.255426] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.267727] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.270065] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.272027] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.275111] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.280609] acpiphp: Slot [2] registered [ 0.282191] acpiphp: Slot [5] registered [ 0.284182] acpiphp: Slot [6] registered [ 0.285133] acpiphp: Slot [7] registered [ 0.287156] acpiphp: Slot [8] registered [ 0.288149] acpiphp: Slot [9] registered [ 0.290172] acpiphp: Slot [10] registered [ 0.291164] acpiphp: Slot [3] registered [ 0.293129] acpiphp: Slot [4] registered [ 0.294137] acpiphp: Slot [11] registered [ 0.295102] acpiphp: Slot [12] registered [ 0.296071] acpiphp: Slot [13] registered [ 0.298102] acpiphp: Slot [14] registered [ 0.299141] acpiphp: Slot [15] registered [ 0.301089] acpiphp: Slot [16] registered [ 0.302109] acpiphp: Slot [17] registered [ 0.304109] acpiphp: Slot [18] registered [ 0.306093] acpiphp: Slot [19] registered [ 0.307095] acpiphp: Slot [20] registered [ 0.308114] acpiphp: Slot [21] registered [ 0.310111] acpiphp: Slot [22] registered [ 0.311097] acpiphp: Slot [23] registered [ 0.312100] acpiphp: Slot [24] registered [ 0.314125] acpiphp: Slot [25] registered [ 0.316168] acpiphp: Slot [26] registered [ 0.317145] acpiphp: Slot [27] registered [ 0.319168] acpiphp: Slot [28] registered [ 0.321131] acpiphp: Slot [29] registered [ 0.322121] acpiphp: Slot [30] registered [ 0.324129] acpiphp: Slot [31] registered [ 0.325217] PCI host bridge to bus 0000:00 [ 0.327027] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.329034] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.331029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.334044] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.336026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.339024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.341174] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.344026] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.348347] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.359013] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.365677] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.368015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.370022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.373018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.376654] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.379848] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.382054] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.384791] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.391024] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.404022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.410015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.415389] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.423023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.429022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.445023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.454585] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.461015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.470017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.490015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.504602] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.509021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.516021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.531022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.545078] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.550022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.556021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.568020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.578090] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.584018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.592014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.616014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.626279] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.631020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.640020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.653034] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.666101] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.668336] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.670333] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.672303] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.674187] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.680047] iommu: Default domain type: Passthrough [ 0.682573] SCSI subsystem initialized [ 0.683145] ACPI: bus type USB registered [ 0.685156] usbcore: registered new interface driver usbfs [ 0.687099] usbcore: registered new interface driver hub [ 0.688103] usbcore: registered new device driver usb [ 0.690221] pps_core: LinuxPPS API ver. 1 registered [ 0.692015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.695100] PTP clock support registered [ 0.697142] EDAC MC: Ver: 3.0.0 [ 0.699157] PCI: Using ACPI for IRQ routing [ 0.702228] NetLabel: Initializing [ 0.703013] NetLabel: domain hash size = 128 [ 0.705010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.707076] NetLabel: unlabeled traffic allowed by default [ 0.710052] vgaarb: loaded [ 0.711347] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.713011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.721776] clocksource: Switched to clocksource kvm-clock [ 0.834430] VFS: Disk quotas dquot_6.6.0 [ 0.836040] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.839030] *** VALIDATE ramfs *** [ 0.840426] *** VALIDATE hugetlbfs *** [ 0.842052] pnp: PnP ACPI init [ 0.844376] pnp: PnP ACPI: found 6 devices [ 0.863703] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.867367] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.869675] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.871897] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.874486] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.877210] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.880245] NET: Registered protocol family 2 [ 0.882920] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.887921] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.891924] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.897475] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.901353] TCP: Hash tables configured (established 65536 bind 65536) [ 0.904572] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.907978] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.910866] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.913903] NET: Registered protocol family 1 [ 0.916965] RPC: Registered named UNIX socket transport module. [ 0.919372] RPC: Registered udp transport module. [ 0.921291] RPC: Registered tcp transport module. [ 0.923158] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.925352] NET: Registered protocol family 44 [ 0.926718] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.929221] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.931375] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.934123] PCI: CLS 0 bytes, default 64 [ 0.936634] Unpacking initramfs... [ 2.334151] debug: unmapping init [mem 0xffff9937bcc54000-0xffff9937bffbffff] [ 2.339230] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.341354] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.344306] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.846161] Initialise system trusted keyrings [ 2.847826] Key type blacklist registered [ 2.849751] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.862696] zbud: loaded [ 2.865829] *** VALIDATE nfs *** [ 2.867216] *** VALIDATE nfs4 *** [ 2.869051] pstore: using deflate compression [ 2.873481] Platform Keyring initialized [ 2.985343] NET: Registered protocol family 38 [ 2.987295] Key type asymmetric registered [ 2.988992] Asymmetric key parser 'x509' registered [ 2.991287] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.994614] io scheduler mq-deadline registered [ 2.996283] io scheduler kyber registered [ 2.998216] io scheduler bfq registered [ 3.000296] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.003451] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.006404] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.009459] ACPI: Power Button [PWRF] [ 3.015167] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.021841] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.039428] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.049963] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.085470] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.113889] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.140398] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.144687] Non-volatile memory driver v1.3 [ 3.146133] Linux agpgart interface v0.103 [ 3.178476] virtio_blk virtio1: [vda] 147000 512-byte logical blocks (75.3 MB/71.8 MiB) [ 3.180729] vda: detected capacity change from 0 to 75264000 [ 3.196061] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.199692] vdb: detected capacity change from 0 to 1073741824 [ 3.213716] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.216032] vdc: detected capacity change from 0 to 2621440000 [ 3.244613] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.247852] vdd: detected capacity change from 0 to 2621440000 [ 3.268395] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.271180] vde: detected capacity change from 0 to 4294967296 [ 3.285748] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.288880] vdf: detected capacity change from 0 to 4294967296 [ 3.296307] libphy: Fixed MDIO Bus: probed [ 3.301638] usbcore: registered new interface driver usbserial_generic [ 3.304129] usbserial: USB Serial support registered for generic [ 3.306841] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.311327] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.313305] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.315585] mousedev: PS/2 mouse device common for all mice [ 3.318401] rtc_cmos 00:05: RTC can wake from S4 [ 3.318420] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.323529] rtc_cmos 00:05: registered as rtc0 [ 3.326790] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.327320] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.330279] intel_pstate: CPU model not supported [ 3.336621] hid: raw HID events driver (C) Jiri Kosina [ 3.338132] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.339182] usbcore: registered new interface driver usbhid [ 3.344068] usbhid: USB HID core driver [ 3.345715] drop_monitor: Initializing network drop monitor service [ 3.348237] Initializing XFRM netlink socket [ 3.350272] NET: Registered protocol family 10 [ 3.353921] Segment Routing with IPv6 [ 3.355306] NET: Registered protocol family 17 [ 3.366178] mpls_gso: MPLS GSO support [ 3.375294] RAS: Correctable Errors collector initialized. [ 3.377032] AVX version of gcm_enc/dec engaged. [ 3.378208] AES CTR mode by8 optimization enabled [ 3.453438] sched_clock: Marking stable (3453415779, 0)->(4360882194, -907466415) [ 3.456635] registered taskstats version 1 [ 3.458643] Loading compiled-in X.509 certificates [ 3.460074] zswap: loaded using pool lzo/zbud [ 3.482963] Key type big_key registered [ 3.493943] Key type encrypted registered [ 3.495318] ima: No TPM chip found, activating TPM-bypass! [ 3.497455] ima: Allocated hash algorithm: sha1 [ 3.499282] ima: No architecture policies found [ 3.501193] evm: Initialising EVM extended attributes: [ 3.503345] evm: security.selinux [ 3.504713] evm: security.ima [ 3.505828] evm: security.capability [ 3.507386] evm: HMAC attrs: 0x1 [ 3.509940] rtc_cmos 00:05: setting system clock to 2026-08-27 20:38:17 UTC (1787863097) [ 3.516286] debug: unmapping init [mem 0xffffffff97e03000-0xffffffff97ffffff] [ 3.519554] debug: unmapping init [mem 0xffffffff96b82000-0xffffffff96e58fff] [ 3.528078] Write protecting the kernel read-only data: 28672k [ 3.531562] debug: unmapping init [mem 0xffffffff95203000-0xffffffff953fffff] [ 3.533572] debug: unmapping init [mem 0xffffffff95b14000-0xffffffff95bfffff] [ 3.568137] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.577094] systemd[1]: Detected virtualization kvm. [ 3.579323] systemd[1]: Detected architecture x86-64. [ 3.581236] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.607065] systemd[1]: No hostname configured. [ 3.608793] systemd[1]: Set hostname to . [ 3.610554] random: systemd: uninitialized urandom read (16 bytes read) [ 3.613018] systemd[1]: Initializing machine ID from random generator. [ 3.657465] random: ln: uninitialized urandom read (6 bytes read) [ 3.758674] random: systemd: uninitialized urandom read (16 bytes read) [ 3.762560] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.766342] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.771784] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.420551] device-mapper: uevent: version 1.0.3 [ 4.422844] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.073630] random: fast init done Starting dracut initqueue hook... [ 5.104978] virtio_net virtio0 ens2: renamed from eth0 [ 5.120424] scsi host0: ata_piix [ 5.137617] scsi host1: ata_piix [ 5.139346] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.141387] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.346173] dracut-initqueue[580]: RTNETLINK answers: File exists [ 10.014801] random: crng init done [ 10.016440] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.355153] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.474647] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.740458] SELinux: Disabled at runtime. [ 11.801589] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.811819] systemd[1]: Detected virtualization kvm. [ 11.813827] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.345298] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.348814] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.353634] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.359266] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.363216] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.370061] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.373742] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice User and Session Slice. Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Huge Pages File System... [ OK ] Reached target Paths. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ 12.495576] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.839389] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.225541] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.251037] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.343956] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.358826] EDAC sbridge: Ver: 1.1.2 [ 15.102688] Key type dns_resolver registered [ 15.415176] NFS: Registering the id_resolver key type [ 15.417276] Key type id_resolver registered [ 15.418881] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg616-server login: [ 49.056198] hrtimer: interrupt took 4267912 ns [ 52.733052] libcfs: loading out-of-tree module taints kernel. [ 52.789510] Key type ._llcrypt registered [ 52.798534] Key type .llcrypt registered [ 53.029492] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [ 74.320656] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 76.400972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 76.417229] alg: No test for adler32 (adler32-zlib) [ 77.803255] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 78.529597] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 80.264166] Key type lgssc registered [ 82.837930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 100.181752] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 143.459312] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 157.190891] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 157.214475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 158.464913] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 158.496778] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 158.619361] Lustre: lustre-MDT0000: new disk, initializing [ 158.713442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 158.731781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 164.000916] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 180.124554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 180.299213] Lustre: 6503:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 180.327608] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 180.331822] Lustre: Skipped 1 previous similar message [ 180.466464] Lustre: lustre-MDT0001: new disk, initializing [ 180.561495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 180.592294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 180.613426] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 185.951684] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 190.998983] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 201.988617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 202.254591] Lustre: lustre-OST0000: new disk, initializing [ 202.265922] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 202.274709] Lustre: 8439:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 202.389344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 204.766787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 204.806086] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 204.935351] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 209.557516] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 223.571981] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 223.731667] Lustre: lustre-OST0001: new disk, initializing [ 223.734386] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 223.741657] Lustre: 9512:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 223.804302] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 230.432193] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 231.961661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 231.970848] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 232.041092] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 243.550793] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 250.905593] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 256.825429] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing check_logdir /tmp/testlogs/ [ 261.451140] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing yml_node [ 265.019870] Lustre: DEBUG MARKER: Client: 2.17.57.108 [ 267.459766] Lustre: DEBUG MARKER: MDS: 2.17.57.108 [ 269.997518] Lustre: DEBUG MARKER: OSS: 2.17.57.108 [ 271.665633] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 16:42:44 EDT 2026 [ 287.711824] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 289.093351] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 308.721133] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 308.733220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 308.748056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 308.768758] Lustre: Skipped 1 previous similar message [ 308.796415] Lustre: Skipped 3 previous similar messages [ 311.493363] Lustre: server umount lustre-MDT0000 complete [ 315.646922] LustreError: 6493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863409 with bad export cookie 12577063092951697456 [ 315.687249] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 315.692157] LustreError: 6493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 318.945357] LustreError: 6510:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 318.948233] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 318.960980] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 318.975630] LustreError: 6510:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 319.007057] Lustre: Skipped 3 previous similar messages [ 320.997381] LustreError: 6509:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 320.998184] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 321.028670] LustreError: 6509:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 321.036502] Lustre: Skipped 2 previous similar messages [ 322.360731] Lustre: server umount lustre-MDT0001 complete [ 326.907571] Lustre: server umount lustre-OST0000 complete [ 330.858311] Lustre: server umount lustre-OST0001 complete [ 337.708497] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [ 347.418539] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 386.365987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 386.684977] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 386.712128] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 386.785052] Lustre: lustre-MDT0000: new disk, initializing [ 386.856654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 386.875672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 391.247917] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 402.527535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 402.711960] Lustre: 15065:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 402.810426] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 402.818091] Lustre: Skipped 1 previous similar message [ 402.973540] Lustre: lustre-MDT0001: new disk, initializing [ 403.129532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 403.170444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 403.189286] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 408.376503] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 419.172577] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 425.570636] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 433.613760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 434.066863] Lustre: lustre-OST0000: new disk, initializing [ 434.073797] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 434.086118] Lustre: 17036:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 434.224335] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 435.330433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 435.342840] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 435.512272] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 440.931412] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 450.436713] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 455.340978] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 455.650527] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 460.889718] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 461.131178] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 466.406092] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 466.406265] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 466.411674] LustreError: Skipped 1 previous similar message [ 466.416307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 466.422184] Lustre: Skipped 1 previous similar message [ 471.529444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 471.541917] Lustre: Skipped 2 previous similar messages [ 477.156028] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 477.380829] Lustre: server umount lustre-OST0000 complete [ 481.767461] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 481.786601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 481.805826] Lustre: Skipped 3 previous similar messages [ 487.723528] Lustre: server umount lustre-MDT0000 complete [ 491.678078] LustreError: 15059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863585 with bad export cookie 12577063092951698989 [ 491.693077] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 491.696266] LustreError: 15059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 492.028045] Lustre: server umount lustre-MDT0001 complete [ 498.143218] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 16:46:30 (1787863590) [ 517.556787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 518.150330] LustreError: 19912:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 518.250783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 522.824948] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 523.235464] LustreError: 19913:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 528.376181] LustreError: 19913:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 531.656769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 536.786783] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 544.531432] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 550.749746] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 552.912996] Lustre: Failing over lustre-MDT0000 [ 553.157039] Lustre: server umount lustre-MDT0000 complete [ 553.444189] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 553.446265] LustreError: 20640:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 553.459195] Lustre: Skipped 2 previous similar messages [ 553.494436] LustreError: 20640:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 569.633103] Lustre: 3634:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787863647/real 1787863647] req@ffff99383f79b480 x1874710413601408/t0(0) o400->MGC192.168.206.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787863663 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 569.649758] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 569.658512] LustreError: 19912:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 569.672355] LustreError: 19912:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 573.969109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 584.070662] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 584.679220] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 584.691935] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 584.718509] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 594.994924] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 600.033111] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 600.035100] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 600.046429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 600.046437] Lustre: Skipped 2 previous similar messages [ 600.102040] Lustre: Skipped 1 previous similar message [ 613.344255] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 613.562854] Lustre: server umount lustre-MDT0000 complete [ 616.929901] LustreError: 20640:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 616.968939] LustreError: 20640:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 617.886426] LustreError: 22572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863711 with bad export cookie 12577063092951700403 [ 617.896852] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 617.907633] LustreError: 22572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 618.388530] Lustre: server umount lustre-MDT0001 complete [ 628.347221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 628.951668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 628.968958] Lustre: Skipped 2 previous similar messages [ 633.960339] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 642.870345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 648.489951] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 657.167806] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 663.616997] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 666.116649] Lustre: Failing over lustre-MDT0000 [ 666.427501] Lustre: server umount lustre-MDT0000 complete [ 668.193569] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 668.211607] Lustre: Skipped 1 previous similar message [ 683.493144] LustreError: 23250:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 683.510475] LustreError: 23250:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 10 previous similar messages [ 684.515770] Lustre: 3634:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787863762/real 1787863762] req@ffff99383cbb9a40 x1874710413631360/t0(0) o400->MGC192.168.206.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787863778 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 684.574182] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 685.294924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 695.779249] LustreError: 3633:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff993704054780 x1874710413634560/t0(0) o250->MGC192.168.206.116@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 696.155239] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 701.411087] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 701.426562] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 701.441928] Lustre: Skipped 1 previous similar message [ 701.443507] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 701.468264] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 711.011891] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 716.770227] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 716.776919] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 716.788832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 716.794099] Lustre: Skipped 7 previous similar messages [ 721.519896] Lustre: server umount lustre-MDT0000 complete [ 725.198758] LustreError: 23229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863819 with bad export cookie 12577063092951701341 [ 725.206533] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 725.216096] LustreError: 23229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 725.528378] Lustre: server umount lustre-MDT0001 complete [ 734.410300] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 16:50:26 (1787863826) [ 751.813684] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [ 763.384147] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 811.124625] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 822.600983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 822.991885] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 823.045330] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 823.115940] Lustre: lustre-MDT0000: new disk, initializing [ 823.269339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 823.283170] Lustre: Skipped 2 previous similar messages [ 823.302359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 827.511804] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 838.285770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 838.396934] Lustre: 29629:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 838.631578] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 843.348803] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 854.722116] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 861.322511] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 871.186694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 871.423420] Lustre: 31596:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 872.671595] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 872.742072] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 872.860689] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 872.886385] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 877.814729] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 887.880892] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 894.425320] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 919.016821] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 919.020115] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 919.023203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 919.023214] Lustre: Skipped 2 previous similar messages [ 919.029221] Lustre: Skipped 2 previous similar messages [ 919.038477] LustreError: Skipped 1 previous similar message [ 930.272411] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 930.569701] Lustre: server umount lustre-OST0000 complete [ 935.906416] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 940.939766] Lustre: server umount lustre-MDT0000 complete [ 944.616838] LustreError: 29639:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 944.642980] LustreError: 29639:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 13 previous similar messages [ 945.065056] LustreError: 29619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864039 with bad export cookie 12577063092951701747 [ 945.093786] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 945.098272] LustreError: 29619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 945.648517] Lustre: server umount lustre-MDT0001 complete [ 957.827527] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 960.920462] Key type lgssc unregistered [ 961.279627] LNet: 33579:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 961.293101] LNetError: 33579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 961.340935] LNet: Removed LNI 192.168.206.116@tcp [ 962.489144] Key type .llcrypt unregistered [ 962.491666] Key type ._llcrypt unregistered [ 989.383100] Key type ._llcrypt registered [ 989.386493] Key type .llcrypt registered [ 989.524346] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 991.097491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 991.176756] alg: No test for adler32 (adler32-zlib) [ 992.316954] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 992.634028] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 994.408432] Key type lgssc registered [ 995.632988] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1004.108911] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1004.129962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1005.721601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1010.636568] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 1015.276690] LustreError: 34996:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1015.305234] LustreError: 34996:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1019.454741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1020.150427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1025.064276] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 1032.200876] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1038.976531] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1046.377706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1046.895308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1051.060629] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1053.928613] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 1062.361589] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1065.353974] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1080.284626] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 16:56:12 (1787864172) [ 1082.575347] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1084.586724] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1091.427095] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 16:56:23 (1787864183) [ 1101.902513] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 16:56:33 (1787864193) [ 1102.956479] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1111.355959] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1111.363346] Lustre: Skipped 2 previous similar messages [ 1113.431032] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 16:56:45 (1787864205) [ 1115.368456] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1115.376574] Lustre: Skipped 2 previous similar messages [ 1125.256788] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1125.262330] Lustre: Skipped 4 previous similar messages [ 1140.815951] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 16:57:12 (1787864232) [ 1144.213089] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1148.120743] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1157.136734] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 16:57:29 (1787864249) [ 1159.048864] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1159.052479] Lustre: Skipped 6 previous similar messages [ 1173.195737] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 16:57:45 (1787864265) [ 1188.324659] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1188.329972] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1188.337948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1188.380015] Lustre: Skipped 1 previous similar message [ 1193.450659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1193.461298] Lustre: Skipped 2 previous similar messages [ 1194.461904] Lustre: server umount lustre-OST0000 complete [ 1198.562666] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1198.571940] Lustre: Skipped 1 previous similar message [ 1198.573192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1198.583234] Lustre: Skipped 1 previous similar message [ 1203.681908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1203.696815] Lustre: Skipped 1 previous similar message [ 1208.807269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1208.823992] Lustre: Skipped 1 previous similar message [ 1212.384190] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1212.539278] Lustre: server umount lustre-MDT0000 complete [ 1213.923668] LustreError: 34991:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1213.941926] LustreError: 34991:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1216.530739] LustreError: 36842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864310 with bad export cookie 855773719230484375 [ 1216.531440] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1216.548640] LustreError: 36842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1217.141184] Lustre: server umount lustre-MDT0001 complete [ 1230.908273] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 1234.398266] Key type lgssc unregistered [ 1234.673915] LNet: 42859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1234.686372] LNetError: 42859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1234.715303] LNet: Removed LNI 192.168.206.116@tcp [ 1235.689372] Key type .llcrypt unregistered [ 1235.693207] Key type ._llcrypt unregistered [ 1255.310575] Key type ._llcrypt registered [ 1255.311849] Key type .llcrypt registered [ 1255.383702] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [ 1270.238259] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 1271.329299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1271.368323] alg: No test for adler32 (adler32-zlib) [ 1272.386311] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 1272.610414] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 1274.306212] Key type lgssc registered [ 1275.272436] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1316.772752] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1316.812189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1318.021177] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1318.075229] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1318.180902] Lustre: lustre-MDT0000: new disk, initializing [ 1318.246144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1318.260208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1322.468372] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 1333.469731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1333.536926] Lustre: 46014:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1333.605729] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1333.608164] Lustre: Skipped 1 previous similar message [ 1333.676977] Lustre: lustre-MDT0001: new disk, initializing [ 1333.798740] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1333.832064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1333.840137] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1338.034421] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 1347.247880] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1352.800509] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1359.899827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1360.291814] Lustre: lustre-OST0000: new disk, initializing [ 1360.302211] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1360.332513] Lustre: 47984:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1360.418982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1366.339547] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 1370.168130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1370.187166] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1370.310652] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1376.358544] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1381.208446] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1381.369771] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1385.865310] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1386.081848] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1391.079382] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1391.079635] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1391.083233] LustreError: Skipped 1 previous similar message [ 1391.084780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1391.093166] Lustre: Skipped 1 previous similar message [ 1391.101691] Lustre: Skipped 1 previous similar message [ 1394.891127] Lustre: server umount lustre-OST0000 complete [ 1400.289524] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1400.297700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1400.315803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1401.317939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1401.319714] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1405.338278] Lustre: server umount lustre-MDT0000 complete [ 1408.929797] LustreError: 46006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864502 with bad export cookie 11805346189785729906 [ 1408.952280] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1409.434840] Lustre: server umount lustre-MDT0001 complete [ 1419.454148] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 17:01:51 (1787864511) [ 1426.267608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1426.809522] LustreError: 50459:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1426.834227] LustreError: 50459:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 1426.913329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1431.083527] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 1432.042557] LustreError: 50460:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1436.130758] LustreError: 50459:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1439.143765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1439.418573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1443.714496] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 1450.882115] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1455.889330] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1462.380280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1462.771942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1470.147723] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 1477.654314] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1481.672870] LustreError: 52987:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1482.586280] Lustre: Modifying parameter general.timeout=1 in log params [ 1484.101945] Lustre: Modifying parameter general.timeout=3 in log params [ 1484.105595] Lustre: Skipped 1 previous similar message [ 1486.459840] Lustre: Modifying parameter general.timeout=6 in log params [ 1486.463981] Lustre: Skipped 2 previous similar messages [ 1490.478675] Lustre: Modifying parameter general.timeout=11 in log params [ 1490.487197] Lustre: Skipped 4 previous similar messages [ 1498.555383] Lustre: Modifying parameter general.timeout=21 in log params [ 1498.561502] Lustre: Skipped 9 previous similar messages [ 1514.753190] Lustre: Modifying parameter general.timeout=40 in log params [ 1514.765843] Lustre: Skipped 18 previous similar messages [ 1546.880786] Lustre: Modifying parameter general.timeout=78 in log params [ 1546.884676] Lustre: Skipped 37 previous similar messages [ 1602.238548] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 17:04:54 (1787864694) [ 1605.717934] Lustre: 59924:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1607.953262] Lustre: 60032:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1611.772393] Lustre: 60243:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1611.789533] Lustre: 60243:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1613.062364] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 in log params [ 1613.068569] Lustre: Skipped 57 previous similar messages [ 1616.167954] Lustre: 60439:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1616.180858] Lustre: 60439:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1639.072985] Lustre: 61807:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1639.081307] LustreError: 61807:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1642.684568] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 17:05:35 (1787864735) [ 1654.753595] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1654.756101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1654.794312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1654.801794] Lustre: Skipped 3 previous similar messages [ 1658.678477] Lustre: server umount lustre-MDT0000 complete [ 1659.877351] LustreError: 51187:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1659.884975] LustreError: 51187:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1661.553783] LustreError: 50439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864755 with bad export cookie 11805346189785730788 [ 1661.557175] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1661.563431] LustreError: 50439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1661.823089] Lustre: server umount lustre-MDT0001 complete [ 1675.447909] Lustre: server umount lustre-OST0000 complete [ 1697.884742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1698.146532] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1698.426616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1701.891970] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 1709.205748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1709.270545] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1709.296842] Lustre: 64341:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1713.256610] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 1719.642252] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1724.579839] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1730.059543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1730.227058] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1735.411651] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 1742.268786] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1743.868481] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1743.875750] Lustre: Skipped 17 previous similar messages [ 1743.918756] Lustre: 66905:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1751.018654] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1751.021353] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1751.035120] Lustre: Skipped 2 previous similar messages [ 1751.037509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1751.066366] Lustre: Skipped 1 previous similar message [ 1761.250801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1761.263085] Lustre: Skipped 3 previous similar messages [ 1765.344621] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1765.655775] Lustre: server umount lustre-OST0000 complete [ 1770.979417] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1770.985771] LustreError: Skipped 1 previous similar message [ 1770.988821] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1771.005296] Lustre: Skipped 1 previous similar message [ 1775.393271] Lustre: server umount lustre-MDT0000 complete [ 1776.613577] LustreError: 66179:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1776.642573] LustreError: 66179:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1779.208522] LustreError: 64334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864873 with bad export cookie 11805346189785733798 [ 1779.212624] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1779.215933] LustreError: 64334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1779.628068] Lustre: server umount lustre-MDT0001 complete [ 1790.737122] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 1793.372815] Key type lgssc unregistered [ 1793.804673] LNet: 68120:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1793.816268] LNetError: 68120:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1793.833737] LNet: Removed LNI 192.168.206.116@tcp [ 1794.697922] Key type .llcrypt unregistered [ 1794.701484] Key type ._llcrypt unregistered [ 1817.145652] Key type ._llcrypt registered [ 1817.148317] Key type .llcrypt registered [ 1817.230586] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 1818.083989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1818.126530] alg: No test for adler32 (adler32-zlib) [ 1819.155977] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 1819.356826] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 1821.016208] Key type lgssc registered [ 1821.981234] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1828.352738] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1828.371351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1829.824889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1833.428125] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 1839.078479] LustreError: 69536:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1839.098342] LustreError: 69536:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1841.055059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1841.721087] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1845.632402] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 1852.608671] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1857.259936] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1863.008216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1863.558865] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1869.574294] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 1878.766932] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1887.900942] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1897.959752] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1897.973547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1897.995145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1900.514100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1900.514859] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1902.919546] Lustre: server umount lustre-MDT0000 complete [ 1905.635609] LustreError: 69536:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1905.651812] LustreError: 69536:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1906.054700] LustreError: 69516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865000 with bad export cookie 5773861737102531642 [ 1906.058464] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1906.073122] LustreError: 69516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1906.352662] Lustre: server umount lustre-MDT0001 complete [ 1920.188069] Lustre: server umount lustre-OST0000 complete [ 1945.125563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1945.344087] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1945.368727] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1945.628426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1950.253715] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 1957.881273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1957.990510] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1958.024149] Lustre: 74406:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1958.405455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1962.070459] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 1969.257569] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1974.801291] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1981.735886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1982.009176] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1982.474812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1988.555365] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 1995.593457] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1997.312717] Lustre: 76982:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1998.946350] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2005.987601] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2005.999827] Lustre: Skipped 1 previous similar message [ 2006.004954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2006.009394] Lustre: Skipped 3 previous similar messages [ 2009.726926] Lustre: server umount lustre-OST0000 complete [ 2014.712611] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2014.730072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2014.748704] Lustre: Skipped 1 previous similar message [ 2014.759637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2019.238512] Lustre: server umount lustre-MDT0000 complete [ 2021.348060] LustreError: 75184:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2021.363200] LustreError: 75184:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2022.598384] LustreError: 76984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865116 with bad export cookie 5773861737102532482 [ 2022.612416] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2022.621856] LustreError: 76984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2022.888792] Lustre: server umount lustre-MDT0001 complete [ 2033.062671] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 2035.017668] Key type lgssc unregistered [ 2035.253312] LNet: 78194:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2035.262196] LNetError: 78194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2035.282645] LNet: Removed LNI 192.168.206.116@tcp [ 2035.941627] Key type .llcrypt unregistered [ 2035.944662] Key type ._llcrypt unregistered [ 2057.095633] Key type ._llcrypt registered [ 2057.097979] Key type .llcrypt registered [ 2057.237937] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 2058.075075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2058.185026] alg: No test for adler32 (adler32-zlib) [ 2059.235413] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 2059.411614] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 2061.048164] Key type lgssc registered [ 2061.895806] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2068.626399] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2068.636841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2069.964186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2073.924306] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 2079.204867] LustreError: 79605:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2079.225490] LustreError: 79605:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2082.167186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2082.728977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2087.390596] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 2095.652603] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2101.511900] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2108.373957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2109.006131] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2115.282618] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 2123.067699] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2132.628187] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2133.395419] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2135.034388] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2135.042588] Lustre: Skipped 1 previous similar message [ 2137.664684] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2137.669445] Lustre: Skipped 2 previous similar messages [ 2141.918329] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2141.924672] Lustre: Skipped 4 previous similar messages [ 2149.947982] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2149.953792] Lustre: Skipped 8 previous similar messages [ 2166.925786] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2166.930494] Lustre: Skipped 18 previous similar messages [ 2199.024417] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2199.030654] Lustre: Skipped 37 previous similar messages [ 2222.924311] Lustre: 87701:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2225.396391] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 17:15:17 (1787865317) [ 2226.887119] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2228.730660] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 17:15:21 (1787865321) [ 2236.392202] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2236.394752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2236.407917] Lustre: Skipped 1 previous similar message [ 2236.408535] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2236.433330] Lustre: Skipped 1 previous similar message [ 2241.507447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2241.519816] Lustre: Skipped 1 previous similar message [ 2246.630484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2246.636590] Lustre: Skipped 1 previous similar message [ 2247.136144] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2247.476992] Lustre: server umount lustre-OST0000 complete [ 2251.745362] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2251.748841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2251.748976] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2251.764841] Lustre: Skipped 1 previous similar message [ 2251.787890] Lustre: Skipped 1 previous similar message [ 2256.871525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2256.883390] Lustre: Skipped 1 previous similar message [ 2257.311205] Lustre: server umount lustre-MDT0000 complete [ 2260.508983] LustreError: 79585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865354 with bad export cookie 5823865186515384351 [ 2260.520311] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2260.845686] Lustre: server umount lustre-MDT0001 complete [ 2274.409120] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 2277.249269] Key type lgssc unregistered [ 2277.672322] LNet: 88958:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2277.684241] LNetError: 88958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2277.705777] LNet: Removed LNI 192.168.206.116@tcp [ 2278.723082] Key type .llcrypt unregistered [ 2278.728287] Key type ._llcrypt unregistered [ 2283.295339] Key type ._llcrypt registered [ 2283.297527] Key type .llcrypt registered [ 2283.382912] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2287.651234] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules [ 2288.573706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2288.664665] alg: No test for adler32 (adler32-zlib) [ 2289.793300] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 2290.006341] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 2291.664201] Key type lgssc registered [ 2292.968737] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2293.880460] LustreError: 90162:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2312.969426] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 2315.992424] LustreError: 90162:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2322.737807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2322.764387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2324.569531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2329.737658] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 2334.693983] LustreError: 91179:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2334.703321] LustreError: 91179:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2339.053587] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 17:17:11 (1787865431) [ 2339.810851] LustreError: 91178:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2343.573938] LustreError: 91210:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2343.585632] LustreError: 91210:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2343.593610] LustreError: 91210:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 2343.980447] Lustre: server umount lustre-MDT0000 complete [ 2358.393392] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 2360.978488] Key type lgssc unregistered [ 2361.255062] LNet: 92707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2361.262894] LNetError: 92707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2362.287509] LNet: Removed LNI 192.168.206.116@tcp [ 2363.240480] Key type .llcrypt unregistered [ 2363.243970] Key type ._llcrypt unregistered [ 2381.206828] Key type ._llcrypt registered [ 2381.210829] Key type .llcrypt registered [ 2381.302500] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 2382.305985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2382.343889] alg: No test for adler32 (adler32-zlib) [ 2383.541765] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 2383.909344] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 2385.592189] Key type lgssc registered [ 2386.660392] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2394.968479] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2394.995358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2396.491910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2401.373626] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 2406.885213] LustreError: 93925:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2406.903386] LustreError: 93925:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2410.058335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2410.666178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2414.773559] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 2422.730155] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2429.183919] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2437.396736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2438.177834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2445.152695] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 2452.878985] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2502.546492] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 17:19:55 (1787865595) [ 2503.997241] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2505.600955] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 17:19:58 (1787865598) [ 2510.821968] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2510.828552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2510.838439] Lustre: Skipped 2 previous similar messages [ 2510.855117] Lustre: Skipped 2 previous similar messages [ 2515.453870] Lustre: server umount lustre-MDT0000 complete [ 2515.939069] LustreError: 93925:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2515.963068] LustreError: 93925:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2518.382079] LustreError: 93905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865612 with bad export cookie 12665105620821179281 [ 2518.391462] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2518.626701] Lustre: server umount lustre-MDT0001 complete [ 2532.535108] Lustre: server umount lustre-OST0000 complete [ 2542.588235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2542.832417] LustreError: 98195:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2542.919779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2547.674772] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 2548.205776] LustreError: 98196:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2552.353435] LustreError: 98195:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2557.581768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2558.430788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2563.780831] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 2571.174445] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2577.296958] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2590.448159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2590.691651] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2590.701207] LustreError: 98184:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2590.744458] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2590.755915] LustreError: 100281:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2590.767347] LustreError: 100281:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2590.775903] LustreError: 100281:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2590.829561] Lustre: server umount lustre-OST0000 complete [ 2590.831847] LustreError: 100281:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2599.854869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2600.077683] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2600.088153] LustreError: 98185:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2600.100122] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2600.111479] LustreError: 100743:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2600.122164] LustreError: 100743:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2600.130257] LustreError: 100743:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2600.217597] Lustre: server umount lustre-OST0000 complete [ 2600.222297] LustreError: 100743:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2610.767132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2611.018107] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2611.026094] Lustre: Found index 0 for lustre-OST0000, updating log [ 2611.044924] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2611.118974] Lustre: lustre-OST0000: new disk, initializing [ 2611.122296] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2611.127680] Lustre: 101278:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2611.431308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2618.556388] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 2622.994570] Lustre: Failing over lustre-OST0000 [ 2623.264516] Lustre: server umount lustre-OST0000 complete [ 2628.066328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2628.072462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2628.093248] Lustre: Skipped 1 previous similar message [ 2628.111672] Lustre: Skipped 1 previous similar message [ 2629.154931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2629.176408] Lustre: Skipped 1 previous similar message [ 2634.211946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2634.221353] Lustre: Skipped 1 previous similar message [ 2637.374683] Lustre: server umount lustre-MDT0000 complete [ 2639.337707] LustreError: 99516:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2639.352882] LustreError: 99516:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2640.942862] LustreError: 98176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865734 with bad export cookie 12665105620821182025 [ 2640.950429] LustreError: 98176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2640.960996] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2641.320799] Lustre: server umount lustre-MDT0001 complete [ 2651.694309] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 17:22:23 (1787865743) [ 2670.897539] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 2681.526279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2681.917433] LustreError: 103624:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2682.102894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2686.973225] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 2698.585987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2705.042522] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 2708.805685] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2719.174028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2720.047184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2720.053528] Lustre: Skipped 1 previous similar message [ 2720.071138] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2725.160160] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2725.177933] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2725.205181] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2726.199678] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:09 [ 2726.219596] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2726.244558] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2726.244692] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2726.254150] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2727.668948] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 2737.682270] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2737.837346] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2737.935410] Lustre: lustre-OST0001: new disk, initializing [ 2737.941496] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2737.949210] Lustre: 105944:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2745.853749] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 2747.506164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2747.526222] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2747.642032] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2754.932277] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2766.743099] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2773.486116] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2773.489198] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2773.517419] Lustre: Skipped 1 previous similar message [ 2783.713425] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2783.723019] Lustre: Skipped 7 previous similar messages [ 2786.273670] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2786.661936] Lustre: server umount lustre-MDT0001 complete [ 2788.857337] LustreError: 103625:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2788.900347] LustreError: 103625:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 2798.712640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2798.811113] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2798.821839] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2798.857888] Lustre: 103614:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2799.647727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2799.666305] Lustre: Skipped 1 previous similar message [ 2800.336672] LustreError: 107487:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2800.353104] LustreError: 107487:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2800.372025] LustreError: 103617:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2800.493716] LustreError: 107495:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.116@tcp: cfg command failed: rc = -17 [ 2800.500129] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2800.500129] [ 2804.729242] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2805.399124] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 2817.614342] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 17:25:09 (1787865909) [ 2840.080617] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 2860.214776] Lustre: 109564:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2884.175961] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2888.157131] Lustre: 110700:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2983.393979] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2983.409667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2983.423590] Lustre: Skipped 1 previous similar message [ 2983.429930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2983.441308] Lustre: Skipped 1 previous similar message [ 2988.890206] Lustre: server umount lustre-MDT0000 complete [ 2989.038240] LustreError: 103620:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2989.058730] LustreError: 103620:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 8 previous similar messages [ 2992.724694] LustreError: 103605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866086 with bad export cookie 12665105620821182956 [ 2992.725183] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2992.734051] LustreError: 103605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 2992.993845] Lustre: server umount lustre-MDT0001 complete [ 3006.490788] Lustre: server umount lustre-OST0000 complete [ 3010.338520] Lustre: 93238:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787866088/real 1787866088] req@ffff993704154000 x1874712830779520/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787866104 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3010.368344] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3010.384137] Lustre: Skipped 3 previous similar messages [ 3010.977663] Lustre: server umount lustre-OST0001 complete [ 3017.097921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3027.455339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3036.501471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3047.045101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3068.504125] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 3079.380906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3079.422240] Lustre: lustre-MDT0000: reset Object Index mappings [ 3080.078446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3084.565731] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 3093.349061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3093.368190] Lustre: lustre-MDT0001: reset Object Index mappings [ 3100.107811] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 3103.587601] Lustre: 115802:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3110.516488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3118.025537] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 3118.331701] LustreError: 116157:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3118.361234] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3118.371990] LustreError: 116157:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 3124.741376] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 3128.019510] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3133.953385] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3133.961425] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3137.494737] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 3145.355927] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3149.045610] Lustre: 117681:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3207.221477] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 17:31:39 (1787866299) [ 3236.324139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3236.347521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3236.360055] Lustre: Skipped 2 previous similar messages [ 3236.372600] Lustre: Skipped 6 previous similar messages [ 3240.157687] Lustre: server umount lustre-MDT0000 complete [ 3243.933256] LustreError: 114636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866337 with bad export cookie 12665105620821230024 [ 3243.934934] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3243.950322] LustreError: 114636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3258.336172] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3258.708901] Lustre: server umount lustre-MDT0001 complete [ 3277.280197] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3277.448168] Lustre: server umount lustre-OST0000 complete [ 3287.472227] Lustre: server umount lustre-OST0001 complete [ 3294.743594] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [ 3306.808992] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 3346.714884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3347.082285] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3347.140531] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3347.309146] Lustre: lustre-MDT0000: new disk, initializing [ 3347.490983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3347.507846] Lustre: Skipped 3 previous similar messages [ 3347.545662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3352.384538] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 3359.692183] Lustre: server umount lustre-MDT0000 complete [ 3369.311765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3369.562283] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3369.574202] Lustre: 121954:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3369.587229] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3374.263580] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 3380.163679] Lustre: server umount lustre-MDT0000 complete [ 3389.994966] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 17:34:42 (1787866482) [ 3391.737420] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3393.771892] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 17:34:46 (1787866486) [ 3415.928678] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 3426.488816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3430.906603] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 3439.639749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3439.820381] Lustre: 124003:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3439.882432] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3439.889631] Lustre: Skipped 1 previous similar message [ 3439.996640] Lustre: lustre-MDT0001: new disk, initializing [ 3440.207401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3440.229399] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3445.473975] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 3450.783553] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3458.503406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3458.887923] Lustre: lustre-OST0000: new disk, initializing [ 3458.905461] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3458.911942] Lustre: 125582:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3460.630923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3460.641699] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3460.823359] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3466.197623] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 3477.273976] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3477.540471] Lustre: lustre-OST0001: new disk, initializing [ 3477.547833] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3477.565196] Lustre: 126453:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3479.463341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3479.477675] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3479.653692] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3486.772219] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 3498.536921] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3502.642492] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3912.882860] Lustre: 132880:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3912.893644] Lustre: 132880:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3912.897953] Lustre: 132880:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3912.901794] Lustre: 132880:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3912.904610] Lustre: 132880:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3912.908162] Lustre: 132880:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4220.517665] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 17:48:31 (1787867311) [ 4238.311865] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4238.330311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4238.345285] Lustre: Skipped 6 previous similar messages [ 4238.382280] Lustre: Skipped 9 previous similar messages [ 4239.038071] Lustre: server umount lustre-MDT0000 complete [ 4243.429599] LustreError: 125576:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4243.464107] LustreError: 125576:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 16 previous similar messages [ 4244.294891] LustreError: 125212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867338 with bad export cookie 12665105620821299639 [ 4244.302464] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4244.319289] LustreError: 125212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4248.546215] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4248.566818] Lustre: Skipped 1 previous similar message [ 4248.567796] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4248.582515] Lustre: Skipped 1 previous similar message [ 4251.374235] Lustre: server umount lustre-MDT0001 complete [ 4263.594388] Lustre: server umount lustre-OST0000 complete [ 4271.576565] Lustre: server umount lustre-OST0001 complete [ 4279.526236] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [ 4290.271268] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 4329.845230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4330.272894] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4330.309849] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4330.412639] Lustre: lustre-MDT0000: new disk, initializing [ 4330.577239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4330.587168] Lustre: Skipped 5 previous similar messages [ 4330.625120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4335.336710] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 4346.514046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4346.653381] Lustre: 135936:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4346.703768] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4346.708104] Lustre: Skipped 1 previous similar message [ 4346.770817] Lustre: lustre-MDT0001: new disk, initializing [ 4346.882213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4346.894134] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4351.036288] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 4360.333316] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4366.122754] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4373.725752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4374.087437] Lustre: lustre-OST0000: new disk, initializing [ 4374.091279] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4374.101096] Lustre: 137909:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4375.529232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4375.536795] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4375.645263] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4380.547269] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 4390.307550] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4401.395627] Lustre: lustre-MDD0000: changelog on [ 4404.314535] Lustre: lustre-MDD0001: changelog on [ 6432.439256] Lustre: lustre-MDD0001: changelog off [ 6434.143610] Lustre: 137904:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6434.813860] Lustre: lustre-MDD0000: changelog off [ 6444.008168] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6444.009070] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6444.020214] Lustre: Skipped 1 previous similar message [ 6444.023721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6444.033575] LustreError: Skipped 1 previous similar message [ 6444.034719] Lustre: Skipped 3 previous similar messages [ 6445.623657] Lustre: server umount lustre-OST0000 complete [ 6449.121945] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6449.138248] Lustre: Skipped 1 previous similar message [ 6449.140770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6454.245440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6454.255515] Lustre: Skipped 2 previous similar messages [ 6455.480507] Lustre: server umount lustre-MDT0000 complete [ 6459.364882] LustreError: 136821:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6459.393200] LustreError: 136821:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 6459.556433] LustreError: 135931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869553 with bad export cookie 12665105620822121404 [ 6459.560331] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6459.569844] LustreError: 135931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6460.020565] Lustre: server umount lustre-MDT0001 complete [ 6470.675822] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 6473.324695] Key type lgssc unregistered [ 6473.630731] LNet: 147077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6473.640865] LNetError: 147077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6474.667213] LNet: Removed LNI 192.168.206.116@tcp [ 6475.436179] Key type .llcrypt unregistered [ 6475.442117] Key type ._llcrypt unregistered [ 6494.375910] Key type ._llcrypt registered [ 6494.380479] Key type .llcrypt registered [ 6494.470828] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [ 6510.154872] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 6511.616325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6511.673883] alg: No test for adler32 (adler32-zlib) [ 6512.661178] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 6512.878896] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 6514.558753] Key type lgssc registered [ 6515.988749] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6554.381047] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6554.472325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6555.871672] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6555.975235] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6556.197816] Lustre: lustre-MDT0000: new disk, initializing [ 6556.331339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6556.373438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6562.335774] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 6574.046140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6574.138594] Lustre: 150278:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6574.245278] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6574.249186] Lustre: Skipped 1 previous similar message [ 6574.380663] Lustre: lustre-MDT0001: new disk, initializing [ 6574.501431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6574.532567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6574.546757] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6578.781489] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 6587.818913] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6593.485303] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6599.758097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6600.045134] Lustre: lustre-OST0000: new disk, initializing [ 6600.056283] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6600.066421] Lustre: 152248:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6600.149725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6606.512530] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 6607.930981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6607.946126] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6608.035712] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6615.003714] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6621.351299] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_module obdecho/obdecho [ 6622.787223] LustreError: 153252:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6622.792638] LustreError: 153252:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6624.327615] LustreError: 153348:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6633.967696] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6633.973511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6633.984402] Lustre: Skipped 2 previous similar messages [ 6633.989925] Lustre: Skipped 2 previous similar messages [ 6639.077451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6639.083316] Lustre: Skipped 2 previous similar messages [ 6640.372732] Lustre: server umount lustre-MDT0000 complete [ 6643.301274] LustreError: 152247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869737 with bad export cookie 12598861579986202515 [ 6643.303826] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6643.311689] LustreError: 152247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6643.527335] Lustre: server umount lustre-MDT0001 complete [ 6657.151378] Lustre: server umount lustre-OST0000 complete [ 6675.251821] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [ 6684.422718] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 6718.036448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6718.213250] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6718.234908] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6718.298856] Lustre: lustre-MDT0000: new disk, initializing [ 6718.363606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6718.380656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6722.423599] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 6734.704301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6734.907905] Lustre: 157063:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6735.052575] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6735.063082] Lustre: Skipped 1 previous similar message [ 6735.308343] Lustre: lustre-MDT0001: new disk, initializing [ 6735.502911] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6735.526368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6735.532284] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6740.017646] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 6750.129240] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6755.582540] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6761.523551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6761.798959] Lustre: lustre-OST0000: new disk, initializing [ 6761.807761] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6761.820360] Lustre: 159031:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6761.912428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6767.964670] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 6769.746539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6769.756676] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6769.845336] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6777.541026] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6781.964383] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6782.139614] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6786.279571] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6786.463624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6790.625907] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6790.627166] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6790.632260] LustreError: Skipped 1 previous similar message [ 6790.640951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6790.663583] Lustre: Skipped 1 previous similar message [ 6790.701335] Lustre: Skipped 1 previous similar message [ 6794.427603] Lustre: server umount lustre-OST0000 complete [ 6800.878189] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6800.879222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6800.902160] Lustre: Skipped 1 previous similar message [ 6806.864462] Lustre: server umount lustre-MDT0000 complete [ 6810.368655] LustreError: 157055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869904 with bad export cookie 12598861579986203474 [ 6810.376929] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6810.381178] LustreError: 157055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6810.842416] Lustre: server umount lustre-MDT0001 complete [ 6814.584371] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 18:31:47 (1787869907) [ 6816.232495] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6817.751228] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 18:31:50 (1787869910) [ 6824.848095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6825.261071] LustreError: 161443:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6825.279149] LustreError: 161443:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 6825.325959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6828.932223] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 6830.568659] LustreError: 161444:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6835.681714] LustreError: 161443:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6835.787979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6840.181975] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 6846.798797] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6852.249346] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6858.523224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6858.790644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6858.796532] Lustre: Skipped 1 previous similar message [ 6865.038940] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 6872.847316] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6880.225719] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6880.229387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6880.251823] Lustre: Skipped 2 previous similar messages [ 6880.268757] Lustre: Skipped 3 previous similar messages [ 6885.230238] Lustre: server umount lustre-MDT0000 complete [ 6885.356361] LustreError: 161444:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6885.370436] LustreError: 161444:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6888.858656] LustreError: 161424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869982 with bad export cookie 12598861579986204356 [ 6888.868524] LustreError: 161424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6888.871339] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6888.965258] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.16@tcp (stopping) [ 6888.971606] Lustre: Skipped 1 previous similar message [ 6890.467932] LustreError: 161440:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6890.472099] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6890.488572] LustreError: 161440:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6898.594100] LustreError: 162497:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6898.611315] LustreError: 162497:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6903.264612] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6903.599391] Lustre: server umount lustre-MDT0001 complete [ 6909.707419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6921.180670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6921.383518] LustreError: 165282:0:(llog.c:1655:llog_backup()) MGC192.168.206.116@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6921.394750] Lustre: 165282:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.116@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6928.354234] LustreError: 165288:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6928.382512] LustreError: 165288:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6961.825263] LustreError: 163273:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6961.844575] LustreError: 163273:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 30 previous similar messages [ 6982.112814] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6982.127935] Lustre: 165282:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.116@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7023.097357] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7027.158279] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 7028.196959] LustreError: 165288:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7028.201378] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7028.223383] LustreError: 165288:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 42 previous similar messages [ 7035.377123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7049.702530] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7059.941829] LustreError: 148502:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004243e577 x1874717160110080/t0(0) o250->MGC192.168.206.116@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7064.839593] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 7065.579207] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7072.672472] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7074.431667] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7080.925743] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7082.780461] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7101.429598] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7101.432868] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7101.444205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7101.452227] Lustre: Skipped 5 previous similar messages [ 7115.232265] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7115.422563] Lustre: server umount lustre-MDT0000 complete [ 7119.040160] LustreError: 161423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870213 with bad export cookie 12598861579986210845 [ 7119.045218] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7119.056019] LustreError: 161423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7119.419462] Lustre: server umount lustre-MDT0001 complete [ 7132.249616] Lustre: server umount lustre-OST0000 complete [ 7151.228916] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [ 7161.198762] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 7198.885303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7199.133429] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7199.166316] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7199.275969] Lustre: lustre-MDT0000: new disk, initializing [ 7199.358770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7199.365671] Lustre: Skipped 1 previous similar message [ 7199.398200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7203.640255] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 7214.065116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7214.183263] Lustre: 170392:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7214.490537] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7219.566167] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 7231.146313] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7237.114633] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7243.443620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7243.785844] Lustre: lustre-OST0000: new disk, initializing [ 7243.791565] Lustre: Skipped 1 previous similar message [ 7243.798616] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7243.804449] Lustre: Skipped 2 previous similar messages [ 7243.815872] Lustre: 172361:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7250.471220] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 7251.524925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7251.542563] Lustre: Skipped 1 previous similar message [ 7251.557321] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7251.721784] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7260.427167] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7265.398379] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7265.627663] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7270.504408] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7270.684403] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7277.550229] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7277.557952] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7277.571712] Lustre: Skipped 2 previous similar messages [ 7277.581230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7277.593516] Lustre: Skipped 9 previous similar messages [ 7278.976402] Lustre: server umount lustre-OST0000 complete [ 7286.241335] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7289.456858] Lustre: server umount lustre-MDT0000 complete [ 7292.897529] LustreError: 170405:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7292.917194] LustreError: 170405:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 32 previous similar messages [ 7293.021196] LustreError: 170385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870387 with bad export cookie 12598861579986227638 [ 7293.029655] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7293.415667] Lustre: server umount lustre-MDT0001 complete [ 7302.452192] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 18:39:54 (1787870394) [ 7309.016331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7314.435931] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 7323.695492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7328.069631] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 7335.142867] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7340.210617] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7346.347492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7352.782502] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 7358.771497] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7368.348944] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 18:41:00 (1787870460) [ 7374.307266] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7374.313500] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7374.330413] Lustre: Skipped 3 previous similar messages [ 7374.333358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7374.338723] Lustre: Skipped 3 previous similar messages [ 7377.600239] Lustre: server umount lustre-OST0000 complete [ 7388.737875] Lustre: server umount lustre-MDT0000 complete [ 7391.789248] LustreError: 174752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870485 with bad export cookie 12598861579986228520 [ 7391.794882] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7391.801314] LustreError: 174752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7392.178910] Lustre: server umount lustre-MDT0001 complete [ 7401.020802] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 7403.289575] Key type lgssc unregistered [ 7403.577626] LNet: 178503:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7403.592978] LNetError: 178503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7403.619601] LNet: Removed LNI 192.168.206.116@tcp [ 7404.464846] Key type .llcrypt unregistered [ 7404.469433] Key type ._llcrypt unregistered [ 7421.949923] Key type ._llcrypt registered [ 7421.952740] Key type .llcrypt registered [ 7422.050930] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 7422.918160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7423.005682] alg: No test for adler32 (adler32-zlib) [ 7424.084605] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 7424.335294] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 7426.010566] Key type lgssc registered [ 7426.838557] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7433.693821] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7433.719513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7449.440540] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7454.560369] LustreError: 179755:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.116@tcp: failed processing log, type 4: rc = -110 [ 7459.680537] LustreError: 179755:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7459.701899] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7459.710714] LustreError: MGC192.168.206.116@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7459.729353] LustreError: 179755:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7459.744465] LustreError: 179755:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7459.753046] LustreError: 179755:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7459.834348] Lustre: server umount lustre-OST0000 complete [ 7459.838674] LustreError: 179755:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7469.399673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7471.140643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7476.231917] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 7481.314875] LustreError: 180255:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7481.332989] LustreError: 180255:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7484.361764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7484.753519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7488.564665] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 7495.404957] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7500.394168] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7506.092489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7506.322330] LustreError: 182071:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7506.330774] Lustre: 182071:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.206.116@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7506.465066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7512.129824] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 7518.183690] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7520.739290] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7520.747118] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7520.766344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7521.762768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7521.766583] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7521.773362] Lustre: Skipped 1 previous similar message [ 7521.787759] Lustre: Skipped 1 previous similar message [ 7526.484127] Lustre: server umount lustre-MDT0000 complete [ 7526.890751] LustreError: 181320:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7526.907754] LustreError: 181320:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7529.773812] LustreError: 180235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870623 with bad export cookie 5101853322191306681 [ 7529.786771] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7529.788454] LustreError: 180235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7530.062902] Lustre: server umount lustre-MDT0001 complete [ 7543.265580] Lustre: server umount lustre-OST0000 complete [ 7552.874900] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [ 7562.732134] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 7593.655219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7593.893313] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7593.928287] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7594.019842] Lustre: lustre-MDT0000: new disk, initializing [ 7594.087245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7594.104437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7598.067747] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 7607.904203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7607.998798] Lustre: 185593:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7608.058245] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7608.067132] Lustre: Skipped 1 previous similar message [ 7608.129996] Lustre: lustre-MDT0001: new disk, initializing [ 7608.210978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7608.245647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7608.261857] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7612.307716] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 7621.002727] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7626.221424] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7632.374161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7632.622984] Lustre: lustre-OST0000: new disk, initializing [ 7632.626226] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7632.631433] Lustre: 187560:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7632.719156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7639.250856] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 7642.241167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7642.249334] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7642.400645] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7648.855708] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7653.918039] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7654.106531] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7657.845667] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7657.996589] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7663.080634] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7663.084660] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7663.085552] LustreError: Skipped 1 previous similar message [ 7663.087430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7663.087435] Lustre: Skipped 1 previous similar message [ 7663.111511] Lustre: Skipped 1 previous similar message [ 7668.201029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7668.206381] Lustre: Skipped 2 previous similar messages [ 7673.318804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7673.325513] Lustre: Skipped 1 previous similar message [ 7674.337201] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7674.489515] Lustre: server umount lustre-OST0000 complete [ 7678.434750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7678.447561] Lustre: Skipped 1 previous similar message [ 7683.570696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7683.576130] Lustre: Skipped 3 previous similar messages [ 7692.256188] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7692.439617] Lustre: server umount lustre-MDT0000 complete [ 7693.795257] LustreError: 186786:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7693.811662] LustreError: 186786:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7695.518382] LustreError: 185587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870789 with bad export cookie 5101853322191307402 [ 7695.520342] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7695.533562] LustreError: 185587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7695.874772] Lustre: server umount lustre-MDT0001 complete [ 7703.598811] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 18:46:36 (1787870796) [ 7719.225358] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 7721.695093] Key type lgssc unregistered [ 7722.036157] LNet: 190512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7722.046546] LNetError: 190512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7722.064783] LNet: Removed LNI 192.168.206.116@tcp [ 7722.943160] Key type .llcrypt unregistered [ 7722.947781] Key type ._llcrypt unregistered [ 7740.747847] Key type ._llcrypt registered [ 7740.750652] Key type .llcrypt registered [ 7740.832587] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 7741.726653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7741.744761] alg: No test for adler32 (adler32-zlib) [ 7742.768640] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 7742.947602] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 7744.608149] Key type lgssc registered [ 7745.712831] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7753.787182] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7753.828122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7755.287494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7759.680205] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 7765.476922] LustreError: 191732:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7765.494846] LustreError: 191732:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7767.533738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7767.947565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7772.590260] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 7780.469236] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7785.384453] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7792.430195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7792.611789] LustreError: 193604:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7792.694162] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7792.696740] LustreError: 193600:0:(llog.c:1696:llog_backup()) MGC192.168.206.116@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7792.702560] Lustre: 193600:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.206.116@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7792.797959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7798.624776] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 7806.068711] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7808.994093] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7808.999160] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7809.014134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7810.531099] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7810.532812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7810.544303] Lustre: Skipped 1 previous similar message [ 7810.563233] Lustre: Skipped 2 previous similar messages [ 7814.449838] Lustre: server umount lustre-MDT0000 complete [ 7815.650813] LustreError: 192744:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7815.673163] LustreError: 192744:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7817.331137] LustreError: 191711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870911 with bad export cookie 9483594000894196576 [ 7817.337531] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7817.340914] LustreError: 191711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7817.635222] Lustre: server umount lustre-MDT0001 complete [ 7830.913341] Lustre: server umount lustre-OST0000 complete [ 7838.768795] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [ 7848.567798] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 7879.848251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7880.120193] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7880.139927] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7880.200395] Lustre: lustre-MDT0000: new disk, initializing [ 7880.285901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7880.299592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7884.028203] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 7893.498748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7893.576310] Lustre: 197124:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7893.606679] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7893.608946] Lustre: Skipped 1 previous similar message [ 7893.611442] LustreError: 197906:0:(llog.c:1696:llog_backup()) MGC192.168.206.116@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7893.616369] LustreError: 197906:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7893.619773] Lustre: 197906:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.116@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7893.632336] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7893.635290] Lustre: Skipped 1 previous similar message [ 7893.700972] Lustre: lustre-MDT0001: new disk, initializing [ 7893.806365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7893.850583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7893.866620] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7898.085783] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 7905.958995] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7911.075237] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7916.833741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7917.016486] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7917.019589] Lustre: Skipped 1 previous similar message [ 7917.022238] LustreError: 199069:0:(llog.c:1696:llog_backup()) MGC192.168.206.116@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7917.028445] LustreError: 199069:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7917.032473] Lustre: 199069:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.116@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7917.038159] Lustre: 199069:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7917.051758] Lustre: lustre-OST0000: new disk, initializing [ 7917.055435] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7917.060719] Lustre: 199093:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7917.130128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7921.525858] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 7925.303160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7925.318040] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7925.382800] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7928.785325] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7932.324721] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7932.485374] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7936.065698] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7936.241177] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7940.578110] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7940.584159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7940.602781] Lustre: Skipped 1 previous similar message [ 7940.614737] Lustre: Skipped 1 previous similar message [ 7945.706507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7945.715374] Lustre: Skipped 1 previous similar message [ 7950.818227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7950.831861] Lustre: Skipped 1 previous similar message [ 7952.352740] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7952.477429] Lustre: server umount lustre-OST0000 complete [ 7955.938482] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7960.546633] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7960.553100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7960.573171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7960.581084] Lustre: Skipped 1 previous similar message [ 7970.272708] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7970.463447] Lustre: server umount lustre-MDT0000 complete [ 7971.307050] LustreError: 197132:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7971.320249] LustreError: 197132:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7973.737314] LustreError: 197117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871067 with bad export cookie 9483594000894197297 [ 7973.738703] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7973.746216] LustreError: 197117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7974.082731] Lustre: server umount lustre-MDT0001 complete [ 7982.351091] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 18:51:14 (1787871074) [ 8002.618643] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 8012.983929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8013.366974] LustreError: 202221:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8013.451396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8016.842948] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 8023.892027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8024.035652] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8024.041229] Lustre: Skipped 4 previous similar messages [ 8027.697577] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 8030.118578] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8036.016380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8041.519069] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 8049.543036] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8049.711299] Lustre: lustre-OST0001: new disk, initializing [ 8049.714261] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8049.718537] Lustre: 204526:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8049.799408] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8049.808679] Lustre: Skipped 2 previous similar messages [ 8054.889548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8054.901234] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8055.017698] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8056.468597] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 8067.151325] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8078.815980] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8080.914726] Lustre: DEBUG MARKER: ADD OST3 [ 8086.280906] loop: module loaded [ 8086.362319] loop0: detected capacity change from 0 to 204800000 [ 8086.531788] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8086.547666] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8086.555932] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8092.702539] loop0: detected capacity change from 0 to 204800000 [ 8093.583244] Lustre: DEBUG MARKER: STOP OST3 [ 8098.339903] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8098.577945] Lustre: lustre-OST0003: new disk, initializing [ 8098.583978] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8098.595609] Lustre: 206524:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8099.925897] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8100.102229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8100.120969] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8100.181612] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8101.987284] LustreError: 206672:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8102.477282] Lustre: Failing over lustre-OST0003 [ 8102.620418] Lustre: server umount lustre-OST0003 complete [ 8105.353811] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 8105.457251] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8105.463112] Lustre: Skipped 1 previous similar message [ 8105.465935] LustreError: 206674:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8105.467972] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8105.473486] LustreError: 206674:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8113.586857] Lustre: DEBUG MARKER: START OST3 again [ 8121.425493] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8121.762127] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8121.772356] Lustre: Skipped 1 previous similar message [ 8121.791378] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8123.301265] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8123.470480] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8123.472835] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8127.983407] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 8148.334643] Lustre: Failing over lustre-OST0003 [ 8148.454251] Lustre: server umount lustre-OST0003 complete [ 8149.483349] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8149.483605] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8149.492281] LustreError: Skipped 1 previous similar message [ 8149.503637] LustreError: 206674:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8149.516404] Lustre: Skipped 1 previous similar message [ 8149.573869] LustreError: 206674:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 12 previous similar messages [ 8167.394105] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8167.406104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8174.811710] Lustre: server umount lustre-MDT0000 complete [ 8177.843597] LustreError: 205293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871271 with bad export cookie 9483594000894198179 [ 8177.846795] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8177.849304] LustreError: 205293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8182.180066] LustreError: 202217:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8182.194698] LustreError: 202217:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 15 previous similar messages [ 8184.354629] Lustre: server umount lustre-MDT0001 complete [ 8188.061971] Lustre: server umount lustre-OST0000 complete [ 8191.999176] Lustre: server umount lustre-OST0001 complete [ 8198.402213] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [ 8207.457773] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 8247.908794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8248.122243] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8248.144340] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8248.241686] Lustre: lustre-MDT0000: new disk, initializing [ 8248.369277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8252.750587] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 8262.840747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8262.972259] Lustre: 211985:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8263.191202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8263.199613] Lustre: Skipped 1 previous similar message [ 8263.263050] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8266.862674] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 8276.576683] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8283.434892] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8291.026795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8291.321648] Lustre: lustre-OST0000: new disk, initializing [ 8291.326497] Lustre: Skipped 1 previous similar message [ 8291.330135] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8291.336273] Lustre: Skipped 2 previous similar messages [ 8291.341434] Lustre: 213957:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8292.912849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8292.923334] Lustre: Skipped 1 previous similar message [ 8292.938458] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8292.995838] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8296.834100] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 8304.615265] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8308.630194] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8308.799757] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8312.958884] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8313.198188] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8318.945835] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8318.950500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8318.970184] Lustre: Skipped 7 previous similar messages [ 8318.972179] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8318.978345] Lustre: Skipped 7 previous similar messages [ 8329.184189] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8329.344790] Lustre: server umount lustre-OST0000 complete [ 8338.775214] Lustre: server umount lustre-MDT0000 complete [ 8339.430887] LustreError: 211992:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8339.451054] LustreError: 211992:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8342.338325] LustreError: 212780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871436 with bad export cookie 9483594000894200475 [ 8342.347023] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8342.363994] LustreError: 212780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8342.681908] Lustre: server umount lustre-MDT0001 complete [ 8346.018587] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 18:57:18 (1787871438) [ 8360.947806] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [ 8370.863126] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 8404.381494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8404.667866] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8404.718603] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8404.839519] Lustre: lustre-MDT0000: new disk, initializing [ 8404.975850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8410.016350] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 8422.116530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8422.258809] Lustre: 218569:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8422.592704] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8427.754445] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 8437.379370] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8442.817767] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8449.789532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8450.228836] Lustre: 220537:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8452.351242] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8457.309801] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 8465.868637] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8469.959338] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8470.197529] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8474.829044] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8475.022807] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8478.182973] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8478.188889] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8478.193614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8478.193621] Lustre: Skipped 6 previous similar messages [ 8478.216625] Lustre: Skipped 3 previous similar messages [ 8490.976128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8491.206579] Lustre: server umount lustre-OST0000 complete [ 8501.896109] Lustre: server umount lustre-MDT0000 complete [ 8503.787707] LustreError: 219408:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8503.812912] LustreError: 219408:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8506.005678] LustreError: 220535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871599 with bad export cookie 9483594000894201357 [ 8506.011840] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8506.015992] LustreError: 220535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8506.648925] Lustre: server umount lustre-MDT0001 complete [ 8516.381993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8522.093551] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 8532.224049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8532.861960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8532.872564] Lustre: Skipped 5 previous similar messages [ 8537.141386] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 8545.124874] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8550.978748] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8558.255682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8565.008341] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 8573.131891] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8583.649304] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8583.660282] LustreError: Skipped 1 previous similar message [ 8588.893044] Lustre: server umount lustre-OST0000 complete [ 8600.517723] Lustre: server umount lustre-MDT0000 complete [ 8604.489895] LustreError: 222736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871698 with bad export cookie 9483594000894202239 [ 8604.494056] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8604.495814] LustreError: 222736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8604.826417] Lustre: server umount lustre-MDT0001 complete [ 8714.116948] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 8716.849600] Key type lgssc unregistered [ 8717.154894] LNet: 226435:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8717.165589] LNetError: 226435:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8717.189151] LNet: Removed LNI 192.168.206.116@tcp [ 8717.975146] Key type .llcrypt unregistered [ 8717.980556] Key type ._llcrypt unregistered [ 8741.010635] Key type ._llcrypt registered [ 8741.013258] Key type .llcrypt registered [ 8741.107836] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [ 8757.453623] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 8758.862767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8758.896973] alg: No test for adler32 (adler32-zlib) [ 8759.986193] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 8760.279034] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 8761.992405] Key type lgssc registered [ 8763.072131] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8803.236135] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8803.258259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8804.539237] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8804.569734] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8804.656869] Lustre: lustre-MDT0000: new disk, initializing [ 8804.727787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8804.739391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8809.825799] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 8821.123938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8821.276639] Lustre: 229790:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8821.343801] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8821.347832] Lustre: Skipped 1 previous similar message [ 8821.465882] Lustre: lustre-MDT0001: new disk, initializing [ 8821.633509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8821.682433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8821.698903] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8827.293239] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 8836.662138] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8841.871325] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8848.781296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8849.019691] Lustre: lustre-OST0000: new disk, initializing [ 8849.022622] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8849.026774] Lustre: 231760:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8849.108177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8854.445635] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 8857.681902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8857.690174] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8857.812881] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8863.549764] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8868.005398] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8868.186477] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8872.331409] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8872.626506] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8878.563786] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8878.565225] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8878.567356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8878.586321] Lustre: Skipped 1 previous similar message [ 8881.047176] Lustre: server umount lustre-OST0000 complete [ 8888.294693] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8888.306375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8888.319167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8888.321148] Lustre: Skipped 1 previous similar message [ 8891.364699] Lustre: server umount lustre-MDT0000 complete [ 8896.117682] LustreError: 230587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871990 with bad export cookie 3905412838812908786 [ 8896.123561] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8896.131160] LustreError: 230587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8896.452432] Lustre: server umount lustre-MDT0001 complete [ 8915.587620] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 8925.728772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8926.204931] LustreError: 234692:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8926.233521] LustreError: 234692:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 8926.282179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8930.664814] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 8931.303787] LustreError: 234693:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8936.421800] LustreError: 234692:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8940.573808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8941.067018] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8946.878096] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 8950.615563] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8957.671793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8958.095380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8964.447460] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 8973.992912] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8974.228334] Lustre: lustre-OST0001: new disk, initializing [ 8974.231092] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8974.237797] Lustre: 237002:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8974.391982] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8980.675966] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 8984.655291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8984.663802] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8984.784492] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8990.838947] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8997.233042] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9010.176753] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 19:08:22 (1787872102) [ 9012.706600] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9012.722506] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9012.733631] Lustre: Skipped 1 previous similar message [ 9012.742657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9012.752402] Lustre: Skipped 1 previous similar message [ 9015.784409] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9015.793347] Lustre: Skipped 1 previous similar message [ 9015.795880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9018.761627] Lustre: server umount lustre-MDT0000 complete [ 9020.920558] LustreError: 234693:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9020.938349] LustreError: 234693:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 9022.755836] LustreError: 234673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872116 with bad export cookie 3905412838812909668 [ 9022.759188] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9022.765625] LustreError: 234673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9023.117223] Lustre: server umount lustre-MDT0001 complete [ 9036.364731] Lustre: server umount lustre-OST0000 complete [ 9050.613842] Lustre: server umount lustre-OST0001 complete [ 9057.856715] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [ 9068.634386] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 9106.759227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9107.071553] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9107.101067] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9107.195419] Lustre: lustre-MDT0000: new disk, initializing [ 9107.276849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9107.296652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9111.991334] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 9124.560351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9124.714791] Lustre: 240949:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9124.773142] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9124.779050] Lustre: Skipped 1 previous similar message [ 9124.889325] Lustre: lustre-MDT0001: new disk, initializing [ 9125.092223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9125.108786] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9129.750425] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 9140.637405] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9147.936282] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9155.332198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9155.726337] Lustre: lustre-OST0000: new disk, initializing [ 9155.733517] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9155.751404] Lustre: 242918:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9157.731768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9157.745363] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9157.857187] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9163.471079] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 9172.900739] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9178.532403] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9178.762754] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9183.623284] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9183.840962] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9188.834528] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9188.849593] Lustre: Skipped 1 previous similar message [ 9188.853784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9188.854699] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9188.859942] Lustre: Skipped 3 previous similar messages [ 9199.079542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9199.087784] Lustre: Skipped 4 previous similar messages [ 9200.097052] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9200.290984] Lustre: server umount lustre-OST0000 complete [ 9204.196194] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9204.208204] Lustre: Skipped 2 previous similar messages [ 9210.571700] Lustre: server umount lustre-MDT0000 complete [ 9214.433676] LustreError: 240961:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9214.447954] LustreError: 240961:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9214.483656] LustreError: 240940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872308 with bad export cookie 3905412838812911803 [ 9214.486333] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9214.495946] LustreError: 240940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9214.826457] Lustre: server umount lustre-MDT0001 complete [ 9223.478257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9223.914639] LustreError: 245134:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9223.988531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9223.995544] Lustre: Skipped 2 previous similar messages [ 9228.383970] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 9237.024848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9241.624933] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 9249.290392] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9255.724777] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9262.802083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9263.017828] LustreError: 246956:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9263.038717] LustreError: 246956:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9269.095481] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 9278.526635] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9427.426055] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9427.435730] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9427.450249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9427.454813] Lustre: Skipped 1 previous similar message [ 9427.468177] Lustre: Skipped 5 previous similar messages [ 9432.543179] Lustre: server umount lustre-OST0000 complete [ 9437.153722] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9437.160553] LustreError: Skipped 1 previous similar message [ 9443.170307] Lustre: server umount lustre-MDT0000 complete [ 9447.690420] LustreError: 245115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872541 with bad export cookie 3905412838812912685 [ 9447.693233] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9447.711830] LustreError: 245115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9448.301705] Lustre: server umount lustre-MDT0001 complete [ 9545.788565] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 9548.430446] Key type lgssc unregistered [ 9548.756272] LNet: 248743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9548.764594] LNetError: 248743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9548.782701] LNet: Removed LNI 192.168.206.116@tcp [ 9549.571159] Key type .llcrypt unregistered [ 9549.572937] Key type ._llcrypt unregistered [ 9575.609111] Key type ._llcrypt registered [ 9575.612537] Key type .llcrypt registered [ 9575.714436] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [ 9591.560353] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 9592.733117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9592.817752] alg: No test for adler32 (adler32-zlib) [ 9593.971849] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 9594.270864] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 9595.960173] Key type lgssc registered [ 9597.339593] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9638.946345] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 9649.073410] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9649.100811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9650.334675] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9650.361990] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9650.447493] Lustre: lustre-MDT0000: new disk, initializing [ 9650.503972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9650.516636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9654.620283] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 9664.678363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9664.773968] Lustre: 252819:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9664.830390] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9664.830473] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9664.837850] Lustre: Skipped 1 previous similar message [ 9664.912235] Lustre: lustre-MDT0001: new disk, initializing [ 9665.004160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9665.063601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9665.073589] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9669.043381] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 9674.124736] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9681.257405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9681.525214] Lustre: lustre-OST0000: new disk, initializing [ 9681.532615] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9681.543400] Lustre: 254450:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9681.671233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9683.520921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9683.534196] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9683.639815] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9687.595864] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 9698.411812] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9698.709803] Lustre: lustre-OST0001: new disk, initializing [ 9698.726577] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9698.738810] Lustre: 255321:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9698.915699] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9707.040121] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 9709.164611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9709.178561] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9709.321608] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9718.889963] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9732.166033] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9745.378081] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9745.386786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9745.396158] Lustre: Skipped 1 previous similar message [ 9745.411914] Lustre: Skipped 3 previous similar messages [ 9745.852899] Lustre: server umount lustre-MDT0000 complete [ 9749.473387] LustreError: 252809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872843 with bad export cookie 6734277380516021292 [ 9749.481740] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9749.490237] LustreError: 252809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9749.761370] Lustre: server umount lustre-MDT0001 complete [ 9763.755317] Lustre: server umount lustre-OST0000 complete [ 9777.298326] Lustre: server umount lustre-OST0001 complete [ 9794.801145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9806.382076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9816.530731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9825.841747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9844.449594] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 9853.930524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9853.951662] Lustre: lustre-MDT0000: reset Object Index mappings [ 9854.390060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9858.659444] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 9859.554779] LustreError: 260333:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9859.580785] LustreError: 260333:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9864.676036] LustreError: 260334:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9866.874758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9866.897501] Lustre: lustre-MDT0001: reset Object Index mappings [ 9867.329577] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9871.869142] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 9874.914202] Lustre: 261475:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9882.788853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9889.290670] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 9892.401263] LustreError: 261830:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9897.449108] LustreError: 261830:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9897.907159] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9898.176716] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9898.181827] Lustre: Skipped 1 previous similar message [ 9899.215523] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9899.225674] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9904.516966] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [ 9913.230347] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9917.060541] Lustre: 263346:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9923.556379] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9923.564075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9923.585293] Lustre: Skipped 2 previous similar messages [ 9923.593182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9924.802341] Lustre: server umount lustre-MDT0000 complete [ 9925.091761] LustreError: 261066:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9925.093947] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9925.112722] LustreError: 261066:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9925.118494] Lustre: Skipped 2 previous similar messages [ 9928.583739] LustreError: 260315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873022 with bad export cookie 6734277380516028082 [ 9928.588575] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9928.597731] LustreError: 260315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9928.857878] Lustre: server umount lustre-MDT0001 complete [ 9943.025736] Lustre: server umount lustre-OST0000 complete [ 9956.357352] Lustre: server umount lustre-OST0001 complete [ 9979.540220] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 19:24:31 (1787873071) [ 9997.670865] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [10008.133320] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [10042.807994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10043.012828] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10043.034838] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10043.089760] Lustre: lustre-MDT0000: new disk, initializing [10043.167756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10043.193656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10047.059810] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [10056.663361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10056.829392] Lustre: 267600:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10056.876734] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10056.880162] Lustre: Skipped 1 previous similar message [10056.933632] Lustre: lustre-MDT0001: new disk, initializing [10057.059944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10057.074042] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10061.366707] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [10070.680519] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10076.461298] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10083.112031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10083.471739] Lustre: lustre-OST0000: new disk, initializing [10083.474788] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10083.480409] Lustre: 269567:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10084.669276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10084.681939] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10084.752641] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10089.373931] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [10099.272211] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10105.945768] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10106.233411] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10110.902631] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10111.232971] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10115.557421] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10115.558756] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10115.560302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10115.574146] Lustre: Skipped 1 previous similar message [10119.529979] Lustre: server umount lustre-OST0000 complete [10123.746708] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10123.760671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10123.786731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10123.800775] Lustre: Skipped 1 previous similar message [10129.531850] Lustre: server umount lustre-MDT0000 complete [10130.918628] LustreError: 267612:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10130.928409] LustreError: 267612:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10133.316239] LustreError: 267593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873227 with bad export cookie 6734277380516030175 [10133.318593] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10133.325940] LustreError: 267593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10133.705889] Lustre: server umount lustre-MDT0001 complete [10153.058859] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [10162.860160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10163.247098] LustreError: 272520:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10163.300643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10163.311214] Lustre: Skipped 2 previous similar messages [10167.993643] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [10176.699420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10181.791372] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [10184.784391] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10192.351102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10198.997639] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [10207.469533] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10207.702168] Lustre: lustre-OST0001: new disk, initializing [10207.715159] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10207.728331] Lustre: 274829:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10209.607834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10209.620544] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10209.739885] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10214.449893] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [10224.177471] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10228.179889] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10233.998938] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10234.293250] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10240.489149] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10240.493700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10240.520524] Lustre: Skipped 4 previous similar messages [10240.535471] Lustre: Skipped 5 previous similar messages [10246.614276] Lustre: server umount lustre-MDT0000 complete [10250.417730] LustreError: 272501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873344 with bad export cookie 6734277380516031057 [10250.433643] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10250.439225] LustreError: 272501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10250.722801] LustreError: 272515:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10250.738442] LustreError: 272515:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10250.807859] Lustre: server umount lustre-MDT0001 complete [10256.395363] Lustre: server umount lustre-OST0000 complete [10260.204561] Lustre: server umount lustre-OST0001 complete [10275.746409] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [10285.893114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10290.598736] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [10299.473973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10299.871061] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10299.877178] Lustre: Skipped 4 previous similar messages [10304.073447] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [10306.958993] Lustre: 279020:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10313.135584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10319.859812] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [10324.859471] LustreError: 279375:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10324.876969] LustreError: 279375:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10329.137059] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10331.461058] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10331.463393] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10336.143463] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [10344.425475] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10347.950370] Lustre: 280896:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10352.789918] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10352.832235] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10363.152441] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 19:30:55 (1787873455) [10381.794036] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10381.811303] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10381.833454] Lustre: Skipped 2 previous similar messages [10381.843298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10381.848453] Lustre: Skipped 4 previous similar messages [10384.411295] Lustre: server umount lustre-MDT0000 complete [10388.472615] LustreError: 277862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873482 with bad export cookie 6734277380516037574 [10388.473812] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10388.482075] LustreError: 277862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10394.989643] Lustre: server umount lustre-MDT0001 complete [10404.995287] Lustre: server umount lustre-OST0000 complete [10408.705786] Lustre: server umount lustre-OST0001 complete [10415.136066] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [10425.315544] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [10473.207454] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [10484.398958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10484.739536] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10484.764888] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10484.876567] Lustre: lustre-MDT0000: new disk, initializing [10484.991866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10489.657909] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [10499.094409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10499.181778] Lustre: 285145:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10499.416782] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10503.200465] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [10513.922340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10514.230087] Lustre: 286777:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10516.107107] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10520.116774] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [10531.092534] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10531.247270] Lustre: 287647:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10533.145571] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10533.153640] Lustre: Skipped 1 previous similar message [10533.252175] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10537.794401] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [10547.011273] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10550.586235] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10550.590682] Lustre: Skipped 1 previous similar message [10565.033661] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 19:34:17 (1787873657) [10569.187190] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10569.192771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10569.202485] Lustre: Skipped 6 previous similar messages [10569.210138] Lustre: Skipped 8 previous similar messages [10569.829485] Lustre: server umount lustre-MDT0000 complete [10574.036109] LustreError: 285137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873668 with bad export cookie 6734277380516045218 [10574.050503] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10574.313108] LustreError: 285152:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10574.337533] LustreError: 285152:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [10574.885710] Lustre: server umount lustre-MDT0001 complete [10581.512209] Lustre: server umount lustre-OST0000 complete [10586.582359] Lustre: server umount lustre-OST0001 complete [10597.072034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10597.851497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10597.855847] Lustre: Skipped 6 previous similar messages [10602.598680] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [10611.479123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10617.637242] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [10625.181877] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10632.374130] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10641.406142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10649.462579] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [10657.523139] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10675.466961] loop1: detected capacity change from 0 to 204800000 [10675.643197] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10675.656561] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10675.676535] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10680.574131] loop1: detected capacity change from 0 to 204800000 [10685.563598] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10685.627395] LustreError: 289878:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs19404: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10685.637066] LustreError: 289878:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs19404 [10685.641027] LustreError: 289878:0:(mgs_handler.c:517:mgs_target_reg()) fs19404-MDT0001: Error checking target index: rc = -13 [10685.646948] LustreError: fs19404-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10685.654836] LustreError: 293609:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10685.661484] LustreError: 293609:0:(tgt_mount.c:2129:server_put_super()) no obd fs19404-MDT0001 [10685.666215] LustreError: 293609:0:(tgt_mount.c:116:server_deregister_mount()) fs19404-MDT0001 not registered [10685.700551] Lustre: server umount fs19404-MDT0001 complete [10685.703931] LustreError: 293609:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10692.889576] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10692.967984] Lustre: Setting parameter fs19404-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs19404-MDT0001 [10693.002932] Lustre: srv-fs19404-MDT0001: No data found on store. Initialize space. [10693.006156] Lustre: Skipped 4 previous similar messages [10693.066332] Lustre: fs19404-MDT0001: new disk, initializing [10693.068434] Lustre: Skipped 3 previous similar messages [10693.193065] Lustre: srv-fs19404-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10693.202773] Lustre: 294078:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs19404-MDT0001-mdtlov: can't create new llog: rc = -115 [10693.208121] LustreError: 294078:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs19404-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10697.137925] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [10703.186514] Lustre: server umount fs19404-MDT0001 complete [10715.183887] loop1: detected capacity change from 0 to 204800000 [10719.922948] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10720.222445] Lustre: 295646:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs19404-MDT0001-mdtlov: can't create new llog: rc = -115 [10720.247101] LustreError: 295646:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs19404-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10722.793457] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10724.145662] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [10729.023447] Lustre: server umount fs19404-MDT0001 complete [10738.158619] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10739.212626] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.16@tcp (stopping) [10739.223533] Lustre: Skipped 2 previous similar messages [10739.681286] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10739.693095] Lustre: Skipped 4 previous similar messages [10745.315127] Lustre: server umount lustre-MDT0000 complete [10753.709341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10753.801310] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10758.239290] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [10759.163488] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10765.142307] loop1: detected capacity change from 0 to 204800000 [10771.056885] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10771.627621] Lustre: 298092:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs19404-MDT0001-mdtlov: can't create new llog: rc = -115 [10771.645234] LustreError: 298092:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs19404-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10775.970241] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [10780.699618] Lustre: server umount fs19404-MDT0001 complete [10804.706273] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10804.713481] LustreError: Skipped 1 previous similar message [10809.334272] Lustre: server umount lustre-OST0000 complete [10827.232128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10827.368979] Lustre: server umount lustre-MDT0000 complete [10830.654687] LustreError: 294623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873924 with bad export cookie 6734277380516047899 [10830.656056] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10830.673307] LustreError: 294623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10831.062620] Lustre: server umount lustre-MDT0001 complete [10842.409622] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [10845.656611] Key type lgssc unregistered [10846.060815] LNet: 300441:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10846.087277] LNetError: 300441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10847.146498] LNet: Removed LNI 192.168.206.116@tcp [10848.196147] Key type .llcrypt unregistered [10848.198800] Key type ._llcrypt unregistered [10866.870169] Key type ._llcrypt registered [10866.874172] Key type .llcrypt registered [10866.967624] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [10867.764322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10867.954791] alg: No test for adler32 (adler32-zlib) [10869.051495] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [10869.306406] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [10870.936310] Key type lgssc registered [10871.879655] Lustre: Echo OBD driver; http://www.lustre.org/ [10879.336559] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10879.357057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10880.896485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10885.976737] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [10890.214829] LustreError: 301714:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10890.240843] LustreError: 301714:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10895.112529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10895.481190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10900.323388] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [10909.268635] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10915.754945] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10922.925217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10923.463070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10925.491086] LustreError: 303543:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10929.886370] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [10930.675630] LustreError: 303543:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10935.793802] LustreError: 303663:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10937.242753] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10940.927870] LustreError: 303544:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10940.972147] LustreError: 303544:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10950.133045] LustreError: 303544:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10950.162377] LustreError: 303544:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10961.176598] loop1: detected capacity change from 0 to 204800000 [10961.366752] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10961.379398] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10961.393279] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10966.524029] LustreError: 303543:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10966.560868] LustreError: 303543:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 10 previous similar messages [10967.008453] loop1: detected capacity change from 0 to 204800000 [10973.161460] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10973.438173] Lustre: fs6002-OST0000: new disk, initializing [10973.441873] Lustre: srv-fs6002-OST0000: No data found on store. Initialize space. [10973.453645] Lustre: 305447:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10973.537373] Lustre: fs6002-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10980.394642] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [10988.868251] Lustre: server umount fs6002-OST0000 complete [11000.183492] loop1: detected capacity change from 0 to 204800000 [11001.344906] LustreError: 303545:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11001.356461] LustreError: 303545:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 18 previous similar messages [11005.368857] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11005.590931] Lustre: fs6002-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11007.463909] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11011.693552] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [11020.582415] loop2: detected capacity change from 0 to 204800000 [11020.841200] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11020.851183] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11020.867986] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11022.841863] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11025.802360] loop2: detected capacity change from 0 to 204800000 [11031.199761] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11031.258652] LustreError: fs6002-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11031.269984] LustreError: fs6002-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11031.277223] LustreError: 308486:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11031.282444] LustreError: 308486:0:(tgt_mount.c:2129:server_put_super()) no obd fs6002-OST0001 [11031.286423] LustreError: 308486:0:(tgt_mount.c:116:server_deregister_mount()) fs6002-OST0001 not registered [11031.315489] Lustre: server umount fs6002-OST0001 complete [11031.318303] LustreError: 308486:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11041.079786] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11041.206452] Lustre: fs6002-OST0001: new disk, initializing [11041.211936] Lustre: srv-fs6002-OST0001: No data found on store. Initialize space. [11041.221749] Lustre: 309037:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11041.302418] Lustre: fs6002-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11048.154080] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [11056.868880] Lustre: server umount fs6002-OST0000 complete [11064.881776] Lustre: server umount fs6002-OST0001 complete [11067.883808] LustreError: 303544:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11067.903638] LustreError: 303544:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 38 previous similar messages [11075.227971] loop1: detected capacity change from 0 to 204800000 [11079.884461] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11080.042811] Lustre: fs6002-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11085.139617] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [11091.007414] loop2: detected capacity change from 0 to 204800000 [11095.998523] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11101.034895] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [11108.862100] Lustre: server umount fs6002-OST0000 complete [11117.703097] Lustre: server umount fs6002-OST0001 complete [11130.341117] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11130.346466] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11130.355501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11130.359606] LustreError: Skipped 1 previous similar message [11130.364589] Lustre: Skipped 1 previous similar message [11135.462216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11135.471668] Lustre: Skipped 1 previous similar message [11140.590573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11140.600519] Lustre: Skipped 1 previous similar message [11142.114393] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11142.301837] Lustre: server umount lustre-OST0000 complete [11145.697382] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11145.705362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11145.713486] Lustre: Skipped 2 previous similar messages [11145.727560] Lustre: Skipped 1 previous similar message [11150.816738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11151.965297] Lustre: server umount lustre-MDT0000 complete [11155.593338] LustreError: 301693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787874249 with bad export cookie 15789558970116574731 [11155.600154] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11155.602774] LustreError: 301693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11156.148076] Lustre: server umount lustre-MDT0001 complete [11165.290515] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [11167.570518] Key type lgssc unregistered [11167.835743] LNet: 314964:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11167.841898] LNetError: 314964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11167.857336] LNet: Removed LNI 192.168.206.116@tcp [11168.547290] Key type .llcrypt unregistered [11168.552681] Key type ._llcrypt unregistered [11197.324931] Key type ._llcrypt registered [11197.328183] Key type .llcrypt registered [11197.438493] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [11212.732861] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [11213.700371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11213.749227] alg: No test for adler32 (adler32-zlib) [11214.815480] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [11215.018391] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [11216.697755] Key type lgssc registered [11217.843738] Lustre: Echo OBD driver; http://www.lustre.org/ [11251.687037] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11251.720190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11252.978877] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11253.021327] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11253.141687] Lustre: lustre-MDT0000: new disk, initializing [11253.259265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11253.277943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11257.572565] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [11268.407579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11268.541645] Lustre: 318821:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11268.598885] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11268.604469] Lustre: Skipped 1 previous similar message [11268.703500] Lustre: lustre-MDT0001: new disk, initializing [11268.805688] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11268.859556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11268.870687] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11273.088876] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [11281.515414] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11285.788918] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11293.966506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11294.401687] Lustre: lustre-OST0000: new disk, initializing [11294.407706] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11294.419562] Lustre: 320791:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11294.683839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11302.337194] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [11304.543336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11304.570444] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11304.780647] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11314.218834] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11316.468112] LustreError: 316994:0:(lmv_obd.c:211:lmv_notify()) activation of lustre-MDT0000_UUID failed: -22 [11316.500739] Lustre: Mounted lustre-client - version 2.17.57_108_ga833322 [11317.558251] Lustre: Unmounted lustre-client [11330.532354] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11330.533330] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11330.541663] LustreError: Skipped 1 previous similar message [11330.548696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11332.067827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11332.077268] Lustre: Skipped 2 previous similar messages [11337.191975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11337.201516] Lustre: Skipped 1 previous similar message [11341.280174] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11341.475903] Lustre: server umount lustre-OST0000 complete [11345.893836] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11345.903088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11345.912296] Lustre: Skipped 1 previous similar message [11345.926803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11347.426785] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11351.747806] Lustre: server umount lustre-MDT0000 complete [11355.767413] LustreError: 319624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787874449 with bad export cookie 6223816684324761807 [11355.772518] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11355.780222] LustreError: 319624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11356.308259] Lustre: server umount lustre-MDT0001 complete [11370.258151] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [11372.705971] LNet: 1 local NIs in recovery (showing 1): 192.168.206.116@tcp [11373.540635] Key type lgssc unregistered [11373.843798] LNet: 322956:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11373.852956] LNetError: 322956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11445.551927] LNet: Removed LNI 192.168.206.116@tcp [11446.872912] Key type .llcrypt unregistered [11446.875741] Key type ._llcrypt unregistered [11467.639565] Key type ._llcrypt registered [11467.643870] Key type .llcrypt registered [11467.757208] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [11483.676031] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [11484.578455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11484.633748] alg: No test for adler32 (adler32-zlib) [11485.647695] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [11485.931380] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [11487.592161] Key type lgssc registered [11488.603119] Lustre: Echo OBD driver; http://www.lustre.org/ [11518.668261] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 19:50:10 (1787874610) [11524.988093] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11525.017352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11526.252330] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11526.292231] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11526.401544] Lustre: lustre-MDT0000: new disk, initializing [11526.539479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11526.565147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11530.899627] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [11540.336465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11540.439614] Lustre: 326179:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11540.499869] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11540.507381] Lustre: Skipped 1 previous similar message [11540.598484] Lustre: lustre-MDT0001: new disk, initializing [11540.712808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11540.758747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11540.769940] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11544.532453] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [11553.597696] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11558.090755] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11564.438568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11564.690781] Lustre: lustre-OST0000: new disk, initializing [11564.694175] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11564.699611] Lustre: 328146:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11564.756102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11569.794701] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [11574.338609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11574.351771] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11574.488369] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11577.779809] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11595.235863] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11595.241238] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11595.246303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11595.248942] LustreError: Skipped 1 previous similar message [11595.273587] Lustre: Skipped 1 previous similar message [11598.050896] Lustre: server umount lustre-OST0000 complete [11602.402413] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11602.411255] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11602.423693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11602.429967] Lustre: Skipped 1 previous similar message [11605.476020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11605.477110] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11610.597989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11610.610899] Lustre: Skipped 2 previous similar messages [11615.720298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11615.727838] Lustre: Skipped 1 previous similar message [11616.224139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11616.523239] Lustre: server umount lustre-MDT0000 complete [11620.103639] LustreError: 326172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787874714 with bad export cookie 12448365283426500175 [11620.104707] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11620.115510] LustreError: 326172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11620.431262] Lustre: server umount lustre-MDT0001 complete [11629.711352] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [11632.294889] Key type lgssc unregistered [11632.668617] LNet: 330058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11632.683897] LNetError: 330058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11632.719485] LNet: Removed LNI 192.168.206.116@tcp [11633.395381] Key type .llcrypt unregistered [11633.401835] Key type ._llcrypt unregistered [11654.719517] Key type ._llcrypt registered [11654.722561] Key type .llcrypt registered [11654.852123] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [11655.944824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11656.086516] alg: No test for adler32 (adler32-zlib) [11657.065316] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [11657.288470] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [11658.984228] Key type lgssc registered [11660.113714] Lustre: Echo OBD driver; http://www.lustre.org/ [11667.585491] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11667.598695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11669.030581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11673.226429] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [11678.187835] LustreError: 331334:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11678.200806] LustreError: 331334:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11682.659185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11683.035963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11687.710962] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [11695.398414] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11701.172487] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11708.043965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11708.406605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11715.130359] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [11721.406310] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11734.625800] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11736.547833] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11743.206293] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11743.207433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11743.229257] Lustre: Skipped 2 previous similar messages [11743.244353] Lustre: Skipped 2 previous similar messages [11744.765040] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.16@tcp (stopping) [11746.826064] Lustre: server umount lustre-MDT0000 complete [11748.322325] LustreError: 333710:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11748.340460] LustreError: 333710:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11749.869894] LustreError: 332068:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11750.760843] LustreError: 331313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787874844 with bad export cookie 12569967032329845133 [11750.770437] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11750.781849] LustreError: 331313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11753.447796] LustreError: 331331:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11753.449370] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11753.482199] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11755.507018] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11755.512462] Lustre: Skipped 1 previous similar message [11760.101258] LustreError: 331329:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11760.109273] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.16@tcp (stopping) [11760.133776] LustreError: 331329:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11765.216329] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11765.493601] Lustre: server umount lustre-MDT0001 complete [11769.690803] Lustre: server umount lustre-OST0000 complete [11779.616402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11785.129669] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [11788.296355] Lustre: 335515:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.116@tcp [11790.989185] Lustre: server umount MGS complete [11801.893793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11802.244692] LustreError: 336087:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11802.264706] LustreError: 336087:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11802.355381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11808.272667] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [11820.526045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11820.584888] LustreError: 336083:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11820.606937] LustreError: 336083:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11821.198769] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11826.092343] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [11834.594057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11835.011573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11841.995704] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [11851.411423] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11853.467375] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11860.159744] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11861.967746] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11867.495064] Lustre: server umount lustre-OST0000 complete [11872.229924] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11872.240046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11872.254893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11872.257404] Lustre: Skipped 2 previous similar messages [11881.565669] Lustre: server umount lustre-MDT0000 complete [11883.490020] LustreError: 336818:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11883.509955] LustreError: 336818:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11884.515782] Lustre: 330646:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787874962/real 1787874962] req@00000000f50b60ac x1874722554968064/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787874978 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11884.543533] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11884.553359] Lustre: Skipped 1 previous similar message [11885.485086] LustreError: 336068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787874979 with bad export cookie 12569967032329845973 [11885.509429] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11885.993495] Lustre: server umount lustre-MDT0001 complete [11895.976702] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [11898.693323] Key type lgssc unregistered [11898.929099] LNet: 339526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11898.935511] LNetError: 339526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11898.954300] LNet: Removed LNI 192.168.206.116@tcp [11899.863940] Key type .llcrypt unregistered [11899.870117] Key type ._llcrypt unregistered [11918.150360] Key type ._llcrypt registered [11918.154536] Key type .llcrypt registered [11918.262546] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [11938.835625] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [11940.558699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11940.725558] alg: No test for adler32 (adler32-zlib) [11941.866744] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [11942.232746] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [11944.033597] Key type lgssc registered [11945.624273] Lustre: Echo OBD driver; http://www.lustre.org/ [11984.694567] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11984.740712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11986.108379] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11986.170575] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11986.289505] Lustre: lustre-MDT0000: new disk, initializing [11986.380409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11986.396097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11991.142297] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [12002.659551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12002.831505] Lustre: 342691:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12002.893601] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12002.903515] Lustre: Skipped 1 previous similar message [12003.025572] Lustre: lustre-MDT0001: new disk, initializing [12003.147433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12003.193745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12003.208335] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12008.257739] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [12017.838303] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12024.314360] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12031.237124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12031.620572] Lustre: lustre-OST0000: new disk, initializing [12031.625900] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12031.636739] Lustre: 344662:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12031.738464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12039.119883] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [12041.780270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12041.788352] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12041.890403] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12048.804783] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12055.340702] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12055.689179] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12061.590640] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12061.905293] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12067.812552] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12067.821471] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12067.828220] LustreError: Skipped 1 previous similar message [12067.832510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12067.882338] Lustre: Skipped 1 previous similar message [12070.705343] Lustre: server umount lustre-OST0000 complete [12074.977977] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12074.984456] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12075.003717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12075.005628] Lustre: Skipped 1 previous similar message [12078.049613] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12078.056215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12078.081454] Lustre: Skipped 1 previous similar message [12080.813182] Lustre: server umount lustre-MDT0000 complete [12084.354265] LustreError: 342682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787875178 with bad export cookie 1248661868290949226 [12084.355137] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12084.363263] LustreError: 342682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12084.767885] Lustre: server umount lustre-MDT0001 complete [12094.982569] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 19:59:47 (1787875187) [12111.430102] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [12122.869843] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [12158.460957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12158.926597] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12158.975050] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12159.075620] Lustre: lustre-MDT0000: new disk, initializing [12159.157301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12159.176138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12166.209301] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [12180.969669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12181.127391] Lustre: 349422:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12181.178434] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12181.182238] Lustre: Skipped 1 previous similar message [12181.253608] Lustre: lustre-MDT0001: new disk, initializing [12181.354901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12181.413959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12181.429944] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12186.442936] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [12199.205350] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12206.787794] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12217.720751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12218.049425] Lustre: lustre-OST0000: new disk, initializing [12218.058632] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12218.067976] Lustre: 351402:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12218.223949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12226.735510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12226.756741] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12226.805687] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [12226.978544] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12238.726690] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12278.243801] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12278.246106] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12278.250584] LustreError: Skipped 1 previous similar message [12278.252219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12278.278680] Lustre: Skipped 1 previous similar message [12279.440344] Lustre: server umount lustre-OST0000 complete [12283.361915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12283.371600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12283.376599] Lustre: Skipped 2 previous similar messages [12293.602014] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12293.609331] Lustre: Skipped 3 previous similar messages [12297.184191] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12297.399475] Lustre: server umount lustre-MDT0000 complete [12298.724794] LustreError: 349433:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12298.750105] LustreError: 349433:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [12300.366656] LustreError: 349412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787875394 with bad export cookie 1248661868290950108 [12300.375030] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12300.379327] LustreError: 349412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12300.660701] Lustre: server umount lustre-MDT0001 complete [12310.014314] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [12312.234413] Key type lgssc unregistered [12312.481383] LNet: 353585:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12312.495048] LNetError: 353585:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12312.512926] LNet: Removed LNI 192.168.206.116@tcp [12313.158468] Key type .llcrypt unregistered [12313.160528] Key type ._llcrypt unregistered [12398.903551] Key type ._llcrypt registered [12398.905184] Key type .llcrypt registered [12399.025119] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [12399.338771] Key type .llcrypt unregistered [12399.344405] Key type ._llcrypt unregistered [12471.761863] Key type ._llcrypt registered [12471.764071] Key type .llcrypt registered [12471.870434] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [12472.150865] Key type .llcrypt unregistered [12472.153938] Key type ._llcrypt unregistered [12499.961411] Key type ._llcrypt registered [12499.963280] Key type .llcrypt registered [12500.050518] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [12500.319291] Key type .llcrypt unregistered [12500.321060] Key type ._llcrypt unregistered [12524.343986] Key type ._llcrypt registered [12524.346949] Key type .llcrypt registered [12524.450572] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [12524.715258] Key type .llcrypt unregistered [12524.718182] Key type ._llcrypt unregistered [12580.192308] Key type ._llcrypt registered [12580.193772] Key type .llcrypt registered [12580.265082] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [12580.427299] Key type .llcrypt unregistered [12580.430106] Key type ._llcrypt unregistered [12605.977477] Key type ._llcrypt registered [12605.982191] Key type .llcrypt registered [12606.126409] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [12606.453600] Key type .llcrypt unregistered [12606.456108] Key type ._llcrypt unregistered [12632.030210] Key type ._llcrypt registered [12632.031773] Key type .llcrypt registered [12632.129397] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [12632.334542] Key type .llcrypt unregistered [12632.336543] Key type ._llcrypt unregistered [12658.994193] Key type ._llcrypt registered [12658.998319] Key type .llcrypt registered [12659.115282] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [12659.409281] Key type .llcrypt unregistered [12659.412334] Key type ._llcrypt unregistered [12684.799864] Key type ._llcrypt registered [12684.803150] Key type .llcrypt registered [12684.975365] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [12685.324497] Key type .llcrypt unregistered [12685.327943] Key type ._llcrypt unregistered [12708.987440] Key type ._llcrypt registered [12708.990345] Key type .llcrypt registered [12709.095309] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [12709.436300] Key type .llcrypt unregistered [12709.439459] Key type ._llcrypt unregistered [12772.073524] Key type ._llcrypt registered [12772.076621] Key type .llcrypt registered [12772.363348] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [12772.958416] Key type .llcrypt unregistered [12772.965212] Key type ._llcrypt unregistered [12806.192575] Key type ._llcrypt registered [12806.203514] Key type .llcrypt registered [12806.340358] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [12806.821298] Key type .llcrypt unregistered [12806.822940] Key type ._llcrypt unregistered [12847.660804] Key type ._llcrypt registered [12847.667655] Key type .llcrypt registered [12847.888795] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [12848.378427] Key type .llcrypt unregistered [12848.388526] Key type ._llcrypt unregistered [12915.360280] Key type ._llcrypt registered [12915.363997] Key type .llcrypt registered [12915.477233] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [12931.460760] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [12932.977050] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12933.064975] alg: No test for adler32 (adler32-zlib) [12934.249351] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [12934.440585] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [12936.144299] Key type lgssc registered [12937.371214] Lustre: Echo OBD driver; http://www.lustre.org/ [12980.293591] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12980.352642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12981.622183] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12981.650762] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12981.758453] Lustre: tst250fs-MDT0000: new disk, initializing [12981.931840] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12981.969701] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12987.096959] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [12998.416932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12998.599620] Lustre: 373812:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12998.678721] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12998.683544] Lustre: Skipped 1 previous similar message [12998.766823] Lustre: tst250fs-MDT0001: new disk, initializing [12998.849838] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12998.893307] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12998.914461] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13003.370980] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [13008.298661] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13015.457797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13015.783237] Lustre: tst250fs-OST0000: new disk, initializing [13015.787530] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13015.798816] Lustre: 375443:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13015.956228] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13020.744611] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13020.758510] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13020.913573] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13022.661167] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [13035.625687] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13035.857910] Lustre: tst250fs-OST0001: new disk, initializing [13035.865490] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13035.878247] Lustre: 376315:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13036.007128] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13042.312261] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13042.323754] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13042.428336] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all [13042.441663] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13047.846702] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13050.516708] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13050.520537] Lustre: Skipped 2 previous similar messages [13070.817981] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13070.831753] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13070.853652] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13073.379969] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13073.382534] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13074.928228] Lustre: server umount tst250fs-MDT0000 complete [13078.245796] LustreError: 373803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787876172 with bad export cookie 11552887682697897599 [13078.246789] LustreError: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13078.261473] LustreError: 373803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13078.499454] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13078.511813] Lustre: Skipped 2 previous similar messages [13078.709547] Lustre: server umount tst250fs-MDT0001 complete [13084.718741] Lustre: server umount tst250fs-OST0000 complete [13088.970387] Lustre: server umount tst250fs-OST0001 complete [13108.234836] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [13115.887801] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [13151.378911] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 20:17:23 (1787876243) [13153.016257] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13171.625281] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [13182.653590] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [13221.378383] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12948 sec ========= 20:18:33 (1787876313) [13223.333504] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 20:18:35 (1787876315) === [13226.916674] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 20:18:39 (1787876319) === [13266.662369] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [13269.313625] Key type lgssc unregistered [13269.569267] LNet: 385466:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13269.582561] LNetError: 385466:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13270.633277] LNet: Removed LNI 192.168.206.116@tcp [13271.404186] Key type .llcrypt unregistered [13271.405833] Key type ._llcrypt unregistered