[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 516296797 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2432MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002329] x2apic enabled [ 0.003023] Switched APIC routing to physical x2apic. [ 0.004000] kvm-guest: setup PV IPIs [ 0.006817] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008014] pid_max: default: 32768 minimum: 301 [ 0.009121] LSM: Security Framework initializing [ 0.010057] Yama: becoming mindful. [ 0.011042] SELinux: Initializing. [ 0.012070] *** VALIDATE selinux *** [ 0.020506] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024763] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025131] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026083] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027110] *** VALIDATE tmpfs *** [ 0.029442] *** VALIDATE proc *** [ 0.031256] *** VALIDATE cgroup *** [ 0.032010] *** VALIDATE cgroup2 *** [ 0.033234] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034141] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036033] Spectre V2 : User space: Vulnerable [ 0.038009] Speculative Store Bypass: Vulnerable [ 0.041525] debug: unmapping init [mem 0xffffffff8ec59000-0xffffffff8ec60fff] [ 0.043191] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044694] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045025] ... version: 2 [ 0.045936] ... bit width: 48 [ 0.046016] ... generic registers: 4 [ 0.047012] ... value mask: 0000ffffffffffff [ 0.048014] ... max period: 00007fffffffffff [ 0.049017] ... fixed-purpose events: 3 [ 0.050013] ... event mask: 000000070000000f [ 0.051303] rcu: Hierarchical SRCU implementation. [ 0.053474] smp: Bringing up secondary CPUs ... [ 0.054598] x86: Booting SMP configuration: [ 0.055027] .... node #0, CPUs: #1 #2 #3 [ 0.059091] smp: Brought up 1 node, 4 CPUs [ 0.061018] smpboot: Max logical packages: 1 [ 0.062012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.234097] node 0 deferred pages initialised in 170ms [ 0.239068] devtmpfs: initialized [ 0.240455] x86/mm: Memory block size: 128MB [ 0.245158] gcov: version magic: 0x41383552 [ 0.249707] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.250097] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.254471] pinctrl core: initialized pinctrl subsystem [ 0.257730] [ 0.258009] ************************************************************* [ 0.261020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.264026] ** ** [ 0.266018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.269021] ** ** [ 0.271019] ** This means that this kernel is built to expose internal ** [ 0.274019] ** IOMMU data structures, which may compromise security on ** [ 0.276017] ** your system. ** [ 0.279021] ** ** [ 0.282035] ** If you see this message and you are not debugging the ** [ 0.284014] ** kernel, report this immediately to your vendor! ** [ 0.286027] ** ** [ 0.289018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.291018] ************************************************************* [ 0.294980] NET: Registered protocol family 16 [ 0.297827] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.302087] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.306084] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.309133] cpuidle: using governor menu [ 0.311930] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.314642] PCI: Using configuration type 1 for base access [ 0.317148] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.327238] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.329040] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.333245] cryptd: max_cpu_qlen set to 1000 [ 0.338250] ACPI: Added _OSI(Module Device) [ 0.340024] ACPI: Added _OSI(Processor Device) [ 0.342017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.344020] ACPI: Added _OSI(Processor Aggregator Device) [ 0.349754] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.359166] ACPI: Interpreter enabled [ 0.360063] ACPI: PM: (supports S0 S3 S4 S5) [ 0.361015] ACPI: Using IOAPIC for interrupt routing [ 0.363273] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.368438] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.382744] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.385052] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.389031] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.392103] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.398695] acpiphp: Slot [2] registered [ 0.400168] acpiphp: Slot [5] registered [ 0.402219] acpiphp: Slot [6] registered [ 0.404240] acpiphp: Slot [7] registered [ 0.406161] acpiphp: Slot [8] registered [ 0.408175] acpiphp: Slot [9] registered [ 0.410219] acpiphp: Slot [10] registered [ 0.412272] acpiphp: Slot [3] registered [ 0.414121] acpiphp: Slot [4] registered [ 0.415129] acpiphp: Slot [11] registered [ 0.417176] acpiphp: Slot [12] registered [ 0.419102] acpiphp: Slot [13] registered [ 0.420085] acpiphp: Slot [14] registered [ 0.422134] acpiphp: Slot [15] registered [ 0.425145] acpiphp: Slot [16] registered [ 0.426321] acpiphp: Slot [17] registered [ 0.429176] acpiphp: Slot [18] registered [ 0.431330] acpiphp: Slot [19] registered [ 0.432098] acpiphp: Slot [20] registered [ 0.433094] acpiphp: Slot [21] registered [ 0.435077] acpiphp: Slot [22] registered [ 0.436065] acpiphp: Slot [23] registered [ 0.437152] acpiphp: Slot [24] registered [ 0.438122] acpiphp: Slot [25] registered [ 0.440115] acpiphp: Slot [26] registered [ 0.442120] acpiphp: Slot [27] registered [ 0.443134] acpiphp: Slot [28] registered [ 0.445076] acpiphp: Slot [29] registered [ 0.447103] acpiphp: Slot [30] registered [ 0.448079] acpiphp: Slot [31] registered [ 0.449081] PCI host bridge to bus 0000:00 [ 0.451028] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.454033] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.456029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.459036] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.463035] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.466033] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.468207] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.471706] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.475542] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.486898] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.492054] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.494032] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.497030] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.501025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.503724] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.507096] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.509049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.512867] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.518021] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.531022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.536584] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.542832] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.555023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.566019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.591029] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.606125] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.616021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.625021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.661019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.674209] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.711020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.719018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.737023] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.749240] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.761020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.770023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.790021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.799364] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.815020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.823021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.840018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.851283] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.859020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.868019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.895018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.908026] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.912475] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.916604] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.919589] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.922317] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.929033] iommu: Default domain type: Passthrough [ 0.933453] SCSI subsystem initialized [ 0.935252] ACPI: bus type USB registered [ 0.937206] usbcore: registered new interface driver usbfs [ 0.939104] usbcore: registered new interface driver hub [ 0.941119] usbcore: registered new device driver usb [ 0.943300] pps_core: LinuxPPS API ver. 1 registered [ 0.945018] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.948090] PTP clock support registered [ 0.951048] EDAC MC: Ver: 3.0.0 [ 0.953163] PCI: Using ACPI for IRQ routing [ 0.956172] NetLabel: Initializing [ 0.957021] NetLabel: domain hash size = 128 [ 0.959017] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.961117] NetLabel: unlabeled traffic allowed by default [ 0.963346] vgaarb: loaded [ 0.965553] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.968025] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.974967] clocksource: Switched to clocksource kvm-clock [ 1.094795] VFS: Disk quotas dquot_6.6.0 [ 1.097626] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.101670] *** VALIDATE ramfs *** [ 1.104331] *** VALIDATE hugetlbfs *** [ 1.106984] pnp: PnP ACPI init [ 1.110662] pnp: PnP ACPI: found 6 devices [ 1.128252] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.132073] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.134525] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.136770] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.139561] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.142183] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.145036] NET: Registered protocol family 2 [ 1.147862] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.153625] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.157723] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.167485] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.171724] TCP: Hash tables configured (established 65536 bind 65536) [ 1.175398] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.181773] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.185659] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.189709] NET: Registered protocol family 1 [ 1.192628] RPC: Registered named UNIX socket transport module. [ 1.195072] RPC: Registered udp transport module. [ 1.196974] RPC: Registered tcp transport module. [ 1.199245] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.202311] NET: Registered protocol family 44 [ 1.204109] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.206496] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.208985] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.211327] PCI: CLS 0 bytes, default 64 [ 1.213210] Unpacking initramfs... [ 2.722456] debug: unmapping init [mem 0xffff94ddfcc54000-0xffff94ddfffbffff] [ 2.728499] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.730788] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.735025] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.271912] Initialise system trusted keyrings [ 3.273509] Key type blacklist registered [ 3.275536] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.293781] zbud: loaded [ 3.301288] *** VALIDATE nfs *** [ 3.302794] *** VALIDATE nfs4 *** [ 3.305070] pstore: using deflate compression [ 3.309791] Platform Keyring initialized [ 3.414968] NET: Registered protocol family 38 [ 3.416827] Key type asymmetric registered [ 3.418501] Asymmetric key parser 'x509' registered [ 3.420380] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.425672] io scheduler mq-deadline registered [ 3.427891] io scheduler kyber registered [ 3.432216] io scheduler bfq registered [ 3.435129] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.440270] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.444692] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.447887] ACPI: Power Button [PWRF] [ 3.453783] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.460891] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.474790] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.482270] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.505553] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.536019] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.567104] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.572684] Non-volatile memory driver v1.3 [ 3.574479] Linux agpgart interface v0.103 [ 3.614388] virtio_blk virtio1: [vda] 139280 512-byte logical blocks (71.3 MB/68.0 MiB) [ 3.617904] vda: detected capacity change from 0 to 71311360 [ 3.694153] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.697804] vdb: detected capacity change from 0 to 1073741824 [ 3.714951] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.718466] vdc: detected capacity change from 0 to 2621440000 [ 3.777045] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.780327] vdd: detected capacity change from 0 to 2621440000 [ 3.795178] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.798540] vde: detected capacity change from 0 to 4294967296 [ 3.814991] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.818684] vdf: detected capacity change from 0 to 4294967296 [ 3.827587] libphy: Fixed MDIO Bus: probed [ 3.836263] usbcore: registered new interface driver usbserial_generic [ 3.840885] usbserial: USB Serial support registered for generic [ 3.845178] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.851663] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.855075] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.858709] mousedev: PS/2 mouse device common for all mice [ 3.863868] rtc_cmos 00:05: RTC can wake from S4 [ 3.870749] rtc_cmos 00:05: registered as rtc0 [ 3.874235] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.879422] intel_pstate: CPU model not supported [ 3.883149] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.884972] hid: raw HID events driver (C) Jiri Kosina [ 3.895745] usbcore: registered new interface driver usbhid [ 3.897389] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.902479] usbhid: USB HID core driver [ 3.909906] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.912154] drop_monitor: Initializing network drop monitor service [ 3.921589] Initializing XFRM netlink socket [ 3.925114] NET: Registered protocol family 10 [ 3.929612] Segment Routing with IPv6 [ 3.932033] NET: Registered protocol family 17 [ 3.935474] mpls_gso: MPLS GSO support [ 3.941423] RAS: Correctable Errors collector initialized. [ 3.944689] AVX version of gcm_enc/dec engaged. [ 3.947446] AES CTR mode by8 optimization enabled [ 4.047389] sched_clock: Marking stable (4047357513, 0)->(4998291526, -950934013) [ 4.052524] registered taskstats version 1 [ 4.055548] Loading compiled-in X.509 certificates [ 4.057725] zswap: loaded using pool lzo/zbud [ 4.083517] Key type big_key registered [ 4.096745] Key type encrypted registered [ 4.099515] ima: No TPM chip found, activating TPM-bypass! [ 4.102980] ima: Allocated hash algorithm: sha1 [ 4.105695] ima: No architecture policies found [ 4.108166] evm: Initialising EVM extended attributes: [ 4.110592] evm: security.selinux [ 4.112329] evm: security.ima [ 4.113731] evm: security.capability [ 4.115832] evm: HMAC attrs: 0x1 [ 4.118637] rtc_cmos 00:05: setting system clock to 2026-06-16 11:21:38 UTC (1781608898) [ 4.126903] debug: unmapping init [mem 0xffffffff8fc03000-0xffffffff8fdfffff] [ 4.131953] debug: unmapping init [mem 0xffffffff8e982000-0xffffffff8ec58fff] [ 4.146176] Write protecting the kernel read-only data: 28672k [ 4.151072] debug: unmapping init [mem 0xffffffff8d003000-0xffffffff8d1fffff] [ 4.155318] debug: unmapping init [mem 0xffffffff8d914000-0xffffffff8d9fffff] [ 4.203731] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.218348] systemd[1]: Detected virtualization kvm. [ 4.222146] systemd[1]: Detected architecture x86-64. [ 4.225613] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.254763] systemd[1]: No hostname configured. [ 4.257201] systemd[1]: Set hostname to . [ 4.259455] random: systemd: uninitialized urandom read (16 bytes read) [ 4.262484] systemd[1]: Initializing machine ID from random generator. [ 4.335897] random: ln: uninitialized urandom read (6 bytes read) [ 4.457692] random: systemd: uninitialized urandom read (16 bytes read) [ 4.462807] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.475996] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.486369] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.456909] device-mapper: uevent: version 1.0.3 [ 5.459936] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon.[ 6.513452] random: fast init done [ 6.547483] virtio_net virtio0 ens2: renamed from eth0 [ 7.376533] scsi host0: ata_piix [ 7.397208] scsi host1: ata_piix [ 7.398884] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.401509] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 11.621991] random: crng init done [ 11.624697] random: 7 urandom warning(s) missed due to ratelimiting [ 12.254313] dracut-initqueue[593]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 13.043023] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.568130] printk: systemd: 22 output lines suppressed due to ratelimiting [ 14.900457] SELinux: Disabled at runtime. [ 14.963501] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 14.973658] systemd[1]: Detected virtualization kvm. [ 14.975725] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 15.650426] systemd[1]: initrd-switch-root.service: Succeeded. [ 15.654381] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 15.665146] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 15.670669] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 15.675895] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 15.697506] systemd[1]: Starting Journal Service... Starting Journal Service... [ 15.705256] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK [0[ 15.781672] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS m] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Reached target Paths. Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ OK ] Started Flush Journal to Persistent Storage. [ 16.573346] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 17.261307] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 17.261800] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 17.576482] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 17.631874] EDAC sbridge: Ver: 1.1.2 [ 20.118979] Key type dns_resolver registered [ 20.500985] NFS: Registering the id_resolver key type [ 20.503455] Key type id_resolver registered [ 20.505154] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Hostname Service... Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg339-server login: [ 50.326064] libcfs: loading out-of-tree module taints kernel. [ 50.347359] Key type ._llcrypt registered [ 50.349097] Key type .llcrypt registered [ 50.430653] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 62.056876] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 62.942202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 62.951230] alg: No test for adler32 (adler32-zlib) [ 64.000661] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 64.441715] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 66.111162] Key type lgssc registered [ 66.848944] Lustre: Echo OBD driver; http://www.lustre.org/ [ 74.940823] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 97.866494] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 106.108760] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 106.131830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 107.318209] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 107.339274] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 107.398824] Lustre: lustre-MDT0000: new disk, initializing [ 107.449715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 107.465393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 109.947630] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 117.819196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 117.876615] Lustre: 6486:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 117.899368] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 117.902582] Lustre: Skipped 1 previous similar message [ 117.951843] Lustre: lustre-MDT0001: new disk, initializing [ 118.008367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 118.026280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 118.031203] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 120.177321] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 123.369029] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 128.270865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 128.414219] Lustre: lustre-OST0000: new disk, initializing [ 128.416553] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 128.420211] Lustre: 8422:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 128.451363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 131.965533] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 132.421544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 132.432487] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 132.482984] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 141.325045] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 141.450102] Lustre: lustre-OST0001: new disk, initializing [ 141.454520] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 141.461307] Lustre: 9494:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 141.506395] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 145.307740] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 150.019322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 150.024909] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 150.055971] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 152.212155] hrtimer: interrupt took 6145974 ns [ 153.453769] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 157.788911] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 161.135267] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing check_logdir /tmp/testlogs/ [ 163.703844] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing yml_node [ 165.907401] Lustre: DEBUG MARKER: Client: 2.17.54.40 [ 167.221401] Lustre: DEBUG MARKER: MDS: 2.17.54.40 [ 168.878206] Lustre: DEBUG MARKER: OSS: 2.17.54.40 [ 169.755576] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Jun 16 07:24:22 EDT 2026 [ 179.214469] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 180.033167] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 195.043480] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 195.050580] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 195.059666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 196.066651] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 196.067117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 196.074817] Lustre: Skipped 1 previous similar message [ 196.084822] Lustre: Skipped 3 previous similar messages [ 199.615404] Lustre: server umount lustre-MDT0000 complete [ 201.799892] LustreError: 6478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781609096 with bad export cookie 17536596110704397827 [ 201.803391] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 201.811053] LustreError: 6478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 206.304697] LustreError: 6493:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 206.304820] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 206.305439] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 206.318226] LustreError: 6493:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 206.339940] Lustre: Skipped 2 previous similar messages [ 208.028496] Lustre: server umount lustre-MDT0001 complete [ 210.845139] Lustre: server umount lustre-OST0000 complete [ 213.342583] Lustre: server umount lustre-OST0001 complete [ 217.049838] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 221.970580] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 241.194325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 241.393443] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 241.412210] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 241.464260] Lustre: lustre-MDT0000: new disk, initializing [ 241.506259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 241.520208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 244.184853] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 250.697788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 250.775379] Lustre: 15035:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 250.839946] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 250.843741] Lustre: Skipped 1 previous similar message [ 250.897479] Lustre: lustre-MDT0001: new disk, initializing [ 250.978060] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 251.008124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 251.015059] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 253.648931] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 259.809677] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 263.286359] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 267.593777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 267.780849] Lustre: lustre-OST0000: new disk, initializing [ 267.784652] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 267.788790] Lustre: 17002:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 267.849361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 269.137994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 269.146383] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 269.200118] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 271.323758] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 277.279621] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 280.040825] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 280.153422] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 282.755900] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 282.883422] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 284.644592] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 284.644928] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 284.653254] LustreError: Skipped 1 previous similar message [ 284.662932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 284.667656] Lustre: Skipped 2 previous similar messages [ 289.771079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 289.775526] Lustre: Skipped 1 previous similar message [ 290.190472] Lustre: server umount lustre-OST0000 complete [ 294.887078] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 294.895482] Lustre: Skipped 2 previous similar messages [ 298.700557] Lustre: server umount lustre-MDT0000 complete [ 300.000532] LustreError: 15044:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 300.018329] LustreError: 15044:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 300.735065] LustreError: 15027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781609195 with bad export cookie 17536596110704399360 [ 300.736210] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 300.743199] LustreError: 15027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 300.911235] Lustre: server umount lustre-MDT0001 complete [ 304.300574] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 07:26:37 (1781609197) [ 315.721852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 316.055309] LustreError: 19872:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 316.100103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 318.785343] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 321.506395] LustreError: 19871:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 324.165452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 326.933816] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 331.384227] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 335.118864] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 336.694220] Lustre: Failing over lustre-MDT0000 [ 336.914024] Lustre: server umount lustre-MDT0000 complete [ 339.943295] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 339.946976] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 339.955218] LustreError: 19866:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 339.955689] Lustre: Skipped 1 previous similar message [ 349.153851] LustreError: 20598:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 349.177218] LustreError: 20598:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 352.769253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 352.842258] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 355.576206] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 358.371352] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 358.375376] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 358.391226] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 361.346859] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 368.613730] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 368.614944] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 368.616779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 368.616784] Lustre: Skipped 2 previous similar messages [ 368.642483] Lustre: Skipped 1 previous similar message [ 378.335297] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 378.451206] Lustre: server umount lustre-MDT0000 complete [ 379.873938] LustreError: 19872:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 379.887374] LustreError: 19872:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 380.990470] LustreError: 19852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781609275 with bad export cookie 17536596110704400774 [ 380.995358] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 381.000661] LustreError: 19852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 381.237493] Lustre: server umount lustre-MDT0001 complete [ 386.295677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 386.622906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 386.632933] Lustre: Skipped 2 previous similar messages [ 389.063406] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 394.301037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 397.170422] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 401.643755] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 405.114141] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 406.406664] Lustre: Failing over lustre-MDT0000 [ 406.533082] Lustre: server umount lustre-MDT0000 complete [ 409.057891] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 409.067114] Lustre: Skipped 1 previous similar message [ 414.176556] LustreError: 23208:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 414.193948] LustreError: 23208:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 421.565416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 421.654141] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 423.923120] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 426.979223] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 426.981992] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 426.995389] Lustre: Skipped 1 previous similar message [ 427.009164] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 429.203076] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 432.099555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 432.102703] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 432.106169] Lustre: Skipped 5 previous similar messages [ 438.077945] Lustre: server umount lustre-MDT0000 complete [ 440.279943] LustreError: 23191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781609334 with bad export cookie 17536596110704401712 [ 440.285878] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 440.288826] LustreError: 23191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 440.537435] Lustre: server umount lustre-MDT0001 complete [ 445.277821] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 07:28:58 (1781609338) [ 453.334773] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 458.103303] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 476.530354] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 482.416970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 482.576848] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 482.614408] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 482.681584] Lustre: lustre-MDT0000: new disk, initializing [ 482.755127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 485.041270] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 490.768036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 490.844418] Lustre: 29568:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 491.004637] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 492.908197] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 497.661812] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 500.563582] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 504.287132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 504.500770] Lustre: 31537:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 505.892762] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 505.931094] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 505.942782] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 507.989274] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 513.251441] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 517.969036] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 529.737252] Lustre: server umount lustre-OST0000 complete [ 531.936619] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 531.941801] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 531.956092] Lustre: Skipped 2 previous similar messages [ 531.959445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 531.966263] Lustre: Skipped 3 previous similar messages [ 541.709291] Lustre: server umount lustre-MDT0000 complete [ 543.515491] LustreError: 32453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781609437 with bad export cookie 17536596110704402118 [ 543.516075] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 543.522756] LustreError: 32453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 553.023974] Lustre: server umount lustre-MDT0001 complete [ 558.439976] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 559.901854] Key type lgssc unregistered [ 560.062678] LNet: 33493:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 560.070565] LNetError: 33493:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 560.083146] LNet: Removed LNI 192.168.203.139@tcp [ 560.568147] Key type .llcrypt unregistered [ 560.569349] Key type ._llcrypt unregistered [ 573.655689] Key type ._llcrypt registered [ 573.657399] Key type .llcrypt registered [ 573.724781] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 574.467529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 574.494794] alg: No test for adler32 (adler32-zlib) [ 575.415664] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 575.544272] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 577.151161] Key type lgssc registered [ 577.677763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 581.972917] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 581.991280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 583.276499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 585.749214] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 590.588551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 590.770443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 592.769151] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 596.253784] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 599.013290] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 602.693065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 602.915147] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 605.972235] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 609.797168] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 616.232970] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 621.614166] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 07:31:54 (1781609514) [ 622.487810] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 623.342991] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 626.517635] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 07:31:59 (1781609519) [ 631.299918] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 07:32:04 (1781609524) [ 631.937594] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 636.245881] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 636.250942] Lustre: Skipped 2 previous similar messages [ 637.034799] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 07:32:10 (1781609530) [ 640.744091] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 640.749738] Lustre: Skipped 4 previous similar messages [ 647.820343] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 07:32:20 (1781609540) [ 649.336689] Lustre: *** cfs_fail_loc=131b, val=2*** [ 651.229600] Lustre: *** cfs_fail_loc=131b, val=2*** [ 655.187185] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 07:32:28 (1781609548) [ 656.087232] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 656.091971] Lustre: Skipped 9 previous similar messages [ 663.184614] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 07:32:36 (1781609556) [ 669.664388] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 669.668665] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 669.675467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 672.739491] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 672.739809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 672.744256] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 675.095966] Lustre: server umount lustre-OST0000 complete [ 677.856758] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 677.857343] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 677.857912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 677.857917] Lustre: Skipped 1 previous similar message [ 677.862889] Lustre: Skipped 1 previous similar message [ 682.866986] Lustre: server umount lustre-MDT0000 complete [ 682.976441] LustreError: 34883:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 682.985729] LustreError: 34883:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 684.652337] LustreError: 34864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781609579 with bad export cookie 13672981697430154761 [ 684.655110] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 684.658426] LustreError: 34864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 684.811443] Lustre: server umount lustre-MDT0001 complete [ 689.499133] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 690.933082] Key type lgssc unregistered [ 691.097410] LNet: 42706:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 691.103280] LNetError: 42706:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 691.119189] LNet: Removed LNI 192.168.203.139@tcp [ 691.513453] Key type .llcrypt unregistered [ 691.515446] Key type ._llcrypt unregistered [ 700.296232] Key type ._llcrypt registered [ 700.298415] Key type .llcrypt registered [ 700.349916] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 708.015886] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 708.421607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 708.451785] alg: No test for adler32 (adler32-zlib) [ 709.351213] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 709.458443] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 711.063891] Key type lgssc registered [ 711.536635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 726.832914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 726.844239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 727.964921] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 727.990166] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 728.031460] Lustre: lustre-MDT0000: new disk, initializing [ 728.081585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 728.096495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 730.068866] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 735.728489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 735.786277] Lustre: 45832:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 735.819328] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 735.824084] Lustre: Skipped 1 previous similar message [ 735.865329] Lustre: lustre-MDT0001: new disk, initializing [ 735.915492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 735.934592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 735.942056] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 737.918781] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 743.016162] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 745.612513] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 748.649051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 748.783691] Lustre: lustre-OST0000: new disk, initializing [ 748.786615] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 748.791816] Lustre: 47800:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 748.836505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 751.651116] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 752.337634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 752.345821] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 752.399953] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 756.577468] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 758.792327] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 758.905552] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 761.067505] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 761.170518] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 762.857126] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 762.858696] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 762.863109] LustreError: Skipped 1 previous similar message [ 762.876552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 762.879917] Lustre: Skipped 1 previous similar message [ 767.969193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 767.975053] Lustre: Skipped 1 previous similar message [ 768.299492] Lustre: server umount lustre-OST0000 complete [ 772.064066] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 772.068095] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 772.078935] Lustre: Skipped 1 previous similar message [ 772.081328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 773.090738] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 778.210908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 778.214889] Lustre: Skipped 3 previous similar messages [ 783.328465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 783.331840] Lustre: Skipped 1 previous similar message [ 784.351126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 784.441378] Lustre: server umount lustre-MDT0000 complete [ 786.261698] LustreError: 46636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781609680 with bad export cookie 5621889845623495806 [ 786.266486] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 786.270286] LustreError: 46636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 786.445616] Lustre: server umount lustre-MDT0001 complete [ 791.361847] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 07:34:44 (1781609684) [ 794.623237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 794.914639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 797.124429] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 800.226848] LustreError: 50259:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 800.235180] LustreError: 50259:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 801.453632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 801.654230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 803.467564] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 806.659567] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 809.140842] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 811.995858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 812.169209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 814.786450] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 818.175720] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 830.582839] LustreError: 52806:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 831.067643] Lustre: Modifying parameter general.timeout=1 in log params [ 832.452607] Lustre: Modifying parameter general.timeout=4 in log params [ 832.460345] Lustre: Skipped 2 previous similar messages [ 834.797227] Lustre: Modifying parameter general.timeout=9 in log params [ 834.800627] Lustre: Skipped 4 previous similar messages [ 839.017748] Lustre: Modifying parameter general.timeout=18 in log params [ 839.021741] Lustre: Skipped 8 previous similar messages [ 847.243148] Lustre: Modifying parameter general.timeout=35 in log params [ 847.246269] Lustre: Skipped 16 previous similar messages [ 863.644347] Lustre: Modifying parameter general.timeout=73 in log params [ 863.648191] Lustre: Skipped 37 previous similar messages [ 893.892978] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 07:36:26 (1781609786) [ 895.535277] Lustre: 59719:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 895.994881] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 in log params [ 896.000508] Lustre: Skipped 59 previous similar messages [ 896.602774] Lustre: 59817:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 898.675957] Lustre: 60014:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 898.682950] Lustre: 60014:0:(mgs_llog.c:1447:mgs_modify_param()) Skipped 1 previous similar message [ 915.071381] Lustre: 61571:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 915.079124] Lustre: 61571:0:(mgs_llog.c:1447:mgs_modify_param()) Skipped 2 previous similar messages [ 915.094781] LustreError: 61571:0:(mgs_handler.c:1202:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 917.257398] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 07:36:50 (1781609810) [ 924.642326] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 924.647302] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 924.655816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 928.851058] Lustre: server umount lustre-MDT0000 complete [ 930.854429] LustreError: 50241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781609825 with bad export cookie 5621889845623496688 [ 930.856957] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 930.860656] LustreError: 50241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 931.025986] Lustre: server umount lustre-MDT0001 complete [ 943.532546] Lustre: server umount lustre-OST0000 complete [ 955.501581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 955.742658] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 955.959041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 960.095252] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 969.590574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 969.732832] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 969.774506] Lustre: 64097:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 974.683255] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 981.887201] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 987.444425] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 993.595343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 993.735860] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 993.952714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 993.965012] Lustre: Skipped 1 previous similar message [ 1000.336210] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1007.748634] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1009.656478] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1009.662955] Lustre: Skipped 20 previous similar messages [ 1009.717863] Lustre: 66664:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1017.831063] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1017.837726] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1017.844788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1017.850444] Lustre: Skipped 3 previous similar messages [ 1017.870804] Lustre: Skipped 4 previous similar messages [ 1022.046775] Lustre: server umount lustre-OST0000 complete [ 1026.534124] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1026.545490] LustreError: Skipped 1 previous similar message [ 1026.549571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1031.573326] Lustre: server umount lustre-MDT0000 complete [ 1033.192572] LustreError: 64108:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1033.208716] LustreError: 64108:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1034.501725] LustreError: 65958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781609928 with bad export cookie 5621889845623499194 [ 1034.507669] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1034.516037] LustreError: 65958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1034.905459] Lustre: server umount lustre-MDT0001 complete [ 1043.658666] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 1046.104696] Key type lgssc unregistered [ 1046.353397] LNet: 67870:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1046.357609] LNetError: 67870:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1046.370596] LNet: Removed LNI 192.168.203.139@tcp [ 1047.216095] Key type .llcrypt unregistered [ 1047.220315] Key type ._llcrypt unregistered [ 1065.825804] Key type ._llcrypt registered [ 1065.835371] Key type .llcrypt registered [ 1065.938512] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 1066.780344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1066.821455] alg: No test for adler32 (adler32-zlib) [ 1067.846920] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 1068.052681] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 1069.727224] Key type lgssc registered [ 1070.651719] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1076.321822] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1076.352414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1077.727954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1080.695155] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1087.024545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1087.555809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1091.034356] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1096.891643] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1100.442993] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1104.925103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1105.278653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1109.419308] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1114.557302] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1130.716341] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1138.150955] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1138.151777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1138.164262] Lustre: Skipped 2 previous similar messages [ 1138.174515] Lustre: Skipped 2 previous similar messages [ 1142.818147] Lustre: server umount lustre-MDT0000 complete [ 1143.264167] LustreError: 69276:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1143.274337] LustreError: 69276:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1145.156734] LustreError: 69256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781610039 with bad export cookie 5979479239692487749 [ 1145.162028] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1145.167767] LustreError: 69256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1145.342407] Lustre: server umount lustre-MDT0001 complete [ 1158.349468] Lustre: server umount lustre-OST0000 complete [ 1176.882692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1177.084238] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1177.108596] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1177.345125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1180.744436] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1187.092541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1187.149041] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1187.163917] Lustre: 74169:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1187.482774] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1190.621777] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1196.436275] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1200.798432] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1206.215504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1206.362372] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1206.664951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1211.232567] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1215.579292] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1222.200828] Lustre: 76744:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1223.352260] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1231.328500] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1231.333354] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1231.335779] LustreError: Skipped 1 previous similar message [ 1231.346726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1231.357097] Lustre: Skipped 1 previous similar message [ 1232.968803] Lustre: server umount lustre-OST0000 complete [ 1236.450232] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1236.460234] Lustre: Skipped 1 previous similar message [ 1236.464728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1236.469050] Lustre: Skipped 1 previous similar message [ 1241.575421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1241.579883] Lustre: Skipped 1 previous similar message [ 1246.689383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1246.698117] Lustre: Skipped 1 previous similar message [ 1250.272525] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1250.389627] Lustre: server umount lustre-MDT0000 complete [ 1250.787252] LustreError: 74179:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1250.814164] LustreError: 74179:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1252.824216] LustreError: 74160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781610147 with bad export cookie 5979479239692488575 [ 1252.825766] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1252.835214] LustreError: 74160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1253.042531] Lustre: server umount lustre-MDT0001 complete [ 1259.579310] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 1261.308855] Key type lgssc unregistered [ 1261.507859] LNet: 77954:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1261.520591] LNetError: 77954:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1261.546740] LNet: Removed LNI 192.168.203.139@tcp [ 1262.140413] Key type .llcrypt unregistered [ 1262.144895] Key type ._llcrypt unregistered [ 1277.345832] Key type ._llcrypt registered [ 1277.348395] Key type .llcrypt registered [ 1277.421438] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 1277.962503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1278.060735] alg: No test for adler32 (adler32-zlib) [ 1279.019068] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 1279.151984] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 1280.767178] Key type lgssc registered [ 1281.497903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1286.356705] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1286.382725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1287.706719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1290.362949] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1296.053429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1296.413207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1299.102380] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1303.967277] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1307.574654] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1311.948568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1312.306538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1316.353087] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1320.572988] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1324.444137] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1325.447234] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1325.453449] Lustre: Skipped 1 previous similar message [ 1326.592360] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1326.596110] Lustre: Skipped 1 previous similar message [ 1328.979631] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1328.982763] Lustre: Skipped 4 previous similar messages [ 1333.135621] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1333.139054] Lustre: Skipped 8 previous similar messages [ 1341.415154] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1341.419710] Lustre: Skipped 17 previous similar messages [ 1357.967426] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1357.970371] Lustre: Skipped 31 previous similar messages [ 1378.638090] Lustre: 87416:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1380.345961] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 07:44:33 (1781610273) [ 1381.226282] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1382.386578] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 07:44:35 (1781610275) [ 1384.929807] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1384.935099] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1384.942692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1388.008072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1388.008148] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1388.009160] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1390.471098] Lustre: server umount lustre-OST0000 complete [ 1393.120494] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1393.121868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1393.127909] Lustre: Skipped 1 previous similar message [ 1393.130153] Lustre: Skipped 1 previous similar message [ 1398.240361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1398.249499] Lustre: Skipped 2 previous similar messages [ 1398.978688] Lustre: server umount lustre-MDT0000 complete [ 1401.265117] LustreError: 79326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781610295 with bad export cookie 8717405466663846668 [ 1401.268923] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1401.273361] LustreError: 79326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1401.459912] Lustre: server umount lustre-MDT0001 complete [ 1408.583357] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 1410.521601] Key type lgssc unregistered [ 1410.725654] LNet: 88667:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1410.730311] LNetError: 88667:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1410.741807] LNet: Removed LNI 192.168.203.139@tcp [ 1411.289928] Key type .llcrypt unregistered [ 1411.294349] Key type ._llcrypt unregistered [ 1414.013967] Key type ._llcrypt registered [ 1414.017137] Key type .llcrypt registered [ 1414.094864] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1417.300749] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules [ 1417.954846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1417.970274] alg: No test for adler32 (adler32-zlib) [ 1418.931781] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 1419.136796] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 1420.807245] Key type lgssc registered [ 1421.633363] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1422.496359] LustreError: 89870:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 1434.846367] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 1441.555654] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1444.092607] LustreError: 89870:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1444.122637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1445.636457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1448.912728] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1454.894170] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 07:45:47 (1781610347) [ 1455.080590] LustreError: 90893:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1455.095567] LustreError: 90893:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1457.963294] LustreError: 90925:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1457.970738] LustreError: 90925:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1457.975965] LustreError: 90925:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 12, retries 0, failed: rc = -5 [ 1458.197804] Lustre: server umount lustre-MDT0000 complete [ 1467.129204] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 1468.970703] Key type lgssc unregistered [ 1469.174562] LNet: 92417:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1469.184495] LNetError: 92417:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1469.201612] LNet: Removed LNI 192.168.203.139@tcp [ 1469.754152] Key type .llcrypt unregistered [ 1469.763377] Key type ._llcrypt unregistered [ 1482.115640] Key type ._llcrypt registered [ 1482.120918] Key type .llcrypt registered [ 1482.227613] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 1483.041555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1483.058272] alg: No test for adler32 (adler32-zlib) [ 1484.054796] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 1484.214083] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 1485.871212] Key type lgssc registered [ 1486.861456] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1491.901723] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1491.915155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1493.329090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1496.215989] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1501.636215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1502.163779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1505.027642] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1509.363920] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1512.490730] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1516.113846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1516.488923] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1520.232512] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1525.277220] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1561.939235] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 07:47:34 (1781610454) [ 1563.232797] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1564.432652] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 07:47:37 (1781610457) [ 1567.725237] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1567.735733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1567.736383] Lustre: Skipped 1 previous similar message [ 1567.748621] Lustre: Skipped 2 previous similar messages [ 1572.833360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1573.049771] Lustre: server umount lustre-MDT0000 complete [ 1575.275893] LustreError: 93596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781610469 with bad export cookie 13039444663508345889 [ 1575.278618] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1575.284752] LustreError: 93596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1575.463529] Lustre: server umount lustre-MDT0001 complete [ 1586.807130] Lustre: server umount lustre-OST0000 complete [ 1594.000601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1594.370613] LustreError: 97882:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1594.386435] LustreError: 97882:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1594.484959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1597.169082] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1599.978386] LustreError: 97883:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1602.488318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1602.898294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1605.314654] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1609.556723] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1612.799901] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1620.087972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1620.200334] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1620.205557] LustreError: 97870:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1620.222418] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1620.226977] LustreError: 99963:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 1620.231041] LustreError: 99963:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 1620.233950] LustreError: 99963:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1620.268237] Lustre: server umount lustre-OST0000 complete [ 1620.272271] LustreError: 99963:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1625.163984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1625.285518] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1625.294055] LustreError: 97870:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1625.304416] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1625.311866] LustreError: 100425:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 1625.316479] LustreError: 100425:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 1625.320687] LustreError: 100425:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1625.356507] Lustre: server umount lustre-OST0000 complete [ 1625.359889] LustreError: 100425:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1630.737052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1630.840394] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1630.843297] Lustre: Found index 0 for lustre-OST0000, updating log [ 1630.849239] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1630.892404] Lustre: lustre-OST0000: new disk, initializing [ 1630.895451] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1630.903770] Lustre: 100960:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1631.048535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1634.200445] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1636.462773] Lustre: Failing over lustre-OST0000 [ 1636.582927] Lustre: server umount lustre-OST0000 complete [ 1638.881248] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1638.887618] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1638.898768] Lustre: Skipped 1 previous similar message [ 1638.902084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1638.906969] Lustre: Skipped 2 previous similar messages [ 1641.444631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1641.444936] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1646.561490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1646.567688] Lustre: Skipped 2 previous similar messages [ 1653.215890] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1653.305216] Lustre: server umount lustre-MDT0000 complete [ 1653.732295] LustreError: 97883:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1653.747771] LustreError: 97883:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1655.526608] LustreError: 97863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781610549 with bad export cookie 13039444663508348633 [ 1655.531347] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1655.534614] LustreError: 97863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1655.716442] Lustre: server umount lustre-MDT0001 complete [ 1660.558115] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 07:49:13 (1781610553) [ 1670.082564] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 1675.710471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1676.033180] LustreError: 103306:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1676.127261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1678.674793] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1681.377672] LustreError: 103307:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1684.243737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1687.856115] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1689.937400] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1694.265168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1698.730073] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1704.270526] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1704.320060] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1704.382622] Lustre: lustre-OST0001: new disk, initializing [ 1704.385943] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1704.392116] Lustre: 105623:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1706.418064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1706.427931] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1706.489669] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1708.581419] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1713.632806] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1718.818345] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1721.826431] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 1721.835351] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1721.848502] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1721.852223] Lustre: Skipped 2 previous similar messages [ 1727.753663] Lustre: server umount lustre-MDT0001 complete [ 1730.018815] LustreError: 103307:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1730.033264] LustreError: 103307:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1733.993304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1734.068955] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1734.076579] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1734.121984] Lustre: 103293:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1734.579346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1734.588828] Lustre: Skipped 3 previous similar messages [ 1735.657708] LustreError: 107212:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1735.661731] LustreError: 107212:0:(tgt_mount.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1735.668893] LustreError: 103298:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1735.743837] LustreError: 107224:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.139@tcp: cfg command failed: rc = -17 [ 1735.748834] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1735.748834] [ 1737.376705] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1739.756380] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1743.713788] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 07:50:36 (1781610636) [ 1755.487862] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 1768.936090] Lustre: 109235:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1783.848687] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1786.375938] Lustre: 110368:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1857.504378] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1857.507535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1857.518822] Lustre: Skipped 3 previous similar messages [ 1857.524922] Lustre: Skipped 4 previous similar messages [ 1857.530866] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1861.895338] Lustre: server umount lustre-MDT0000 complete [ 1862.626348] LustreError: 103301:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1862.636202] LustreError: 103301:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1864.027451] LustreError: 105622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781610758 with bad export cookie 13039444663508349543 [ 1864.027977] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1864.033277] LustreError: 105622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1864.196970] Lustre: server umount lustre-MDT0001 complete [ 1876.036335] Lustre: server umount lustre-OST0000 complete [ 1888.339947] Lustre: server umount lustre-OST0001 complete [ 1891.736155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1897.800358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1902.585669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1908.143559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1920.108281] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 1926.121679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1926.137756] Lustre: lustre-MDT0000: reset Object Index mappings [ 1926.423442] LustreError: 114323:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1926.492909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1929.051743] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1934.347604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1934.359537] Lustre: lustre-MDT0001: reset Object Index mappings [ 1937.643834] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1939.606287] Lustre: 115470:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1943.728885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1948.387780] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1949.688489] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 1950.702450] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 1953.765713] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1957.743874] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1959.405768] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:225) [ 1959.412063] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 1962.759391] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1964.870455] Lustre: 117341:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1998.309584] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 07:54:51 (1781610891) [ 2025.956285] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2025.958980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2025.971535] Lustre: Skipped 4 previous similar messages [ 2025.975747] Lustre: Skipped 4 previous similar messages [ 2040.287140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2040.436385] Lustre: server umount lustre-MDT0000 complete [ 2041.315768] LustreError: 114318:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2041.333503] LustreError: 114318:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 2042.668117] LustreError: 114305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781610937 with bad export cookie 13039444663508396415 [ 2042.671253] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2042.677213] LustreError: 114305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2046.432629] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2046.446868] Lustre: Skipped 2 previous similar messages [ 2049.018099] Lustre: server umount lustre-MDT0001 complete [ 2057.621551] Lustre: server umount lustre-OST0000 complete [ 2066.257665] Lustre: server umount lustre-OST0001 complete [ 2070.075798] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 2075.922190] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 2096.989112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2097.180416] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2097.207953] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2097.279976] Lustre: lustre-MDT0000: new disk, initializing [ 2097.385227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2097.388626] Lustre: Skipped 3 previous similar messages [ 2097.399749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2099.874639] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2104.500978] Lustre: server umount lustre-MDT0000 complete [ 2110.345688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2110.569344] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2110.578469] Lustre: 121605:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2110.591658] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2113.661874] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2116.694248] Lustre: server umount lustre-MDT0000 complete [ 2120.876427] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 07:56:54 (1781611014) [ 2121.766811] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2122.786482] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 07:56:55 (1781611015) [ 2132.788765] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 2138.402112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2141.384916] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2146.287189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2146.411641] Lustre: 123653:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2146.449530] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2146.452855] Lustre: Skipped 1 previous similar message [ 2146.507044] Lustre: lustre-MDT0001: new disk, initializing [ 2146.601372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2146.612127] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2149.544810] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2153.198152] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2157.474373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2157.660759] Lustre: lustre-OST0000: new disk, initializing [ 2157.663498] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2157.671302] Lustre: 125229:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2158.953961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2158.960481] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2159.051983] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2161.824274] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2168.820368] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2168.951543] Lustre: lustre-OST0001: new disk, initializing [ 2168.956616] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2168.970723] Lustre: 126101:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2170.685085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2170.701352] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2170.762785] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2172.935085] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2179.359271] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2181.474908] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2430.726514] Lustre: 132505:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 2430.732617] Lustre: 132505:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 2430.737371] Lustre: 132505:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 2430.742574] Lustre: 132505:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 2430.747960] Lustre: 132505:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 2430.753620] Lustre: 132505:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2594.261274] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 08:04:46 (1781611486) [ 2607.098910] Lustre: server umount lustre-MDT0000 complete [ 2607.586668] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2607.592975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2607.608330] LustreError: 126915:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2607.621564] LustreError: 126915:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2609.075889] LustreError: 132527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781611503 with bad export cookie 13039444663508459352 [ 2609.079436] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2609.085850] LustreError: 132527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2611.172605] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2611.176968] Lustre: Skipped 13 previous similar messages [ 2615.394073] Lustre: server umount lustre-MDT0001 complete [ 2623.571810] Lustre: server umount lustre-OST0000 complete [ 2631.799381] Lustre: server umount lustre-OST0001 complete [ 2635.275238] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 2640.655613] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 2660.581467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2660.712867] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2660.731972] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2660.778909] Lustre: lustre-MDT0000: new disk, initializing [ 2660.826665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2660.830611] Lustre: Skipped 5 previous similar messages [ 2660.843808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2662.947326] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2668.925918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2669.013552] Lustre: 135532:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2669.204983] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2671.594288] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2677.506714] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2680.771613] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2684.760161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2684.926965] Lustre: 137502:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2686.034572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2686.040871] Lustre: Skipped 1 previous similar message [ 2686.044257] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2686.128110] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2688.171734] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2693.539827] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2698.422479] Lustre: lustre-MDD0000: changelog on [ 2699.851376] Lustre: lustre-MDD0001: changelog on [ 3715.039766] Lustre: lustre-MDD0001: changelog off [ 3715.748111] Lustre: 138945:0:(llog_cat.c:977:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 3716.113074] Lustre: lustre-MDD0000: changelog off [ 3719.648269] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3719.651996] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3719.659664] Lustre: Skipped 4 previous similar messages [ 3719.662104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3719.665566] Lustre: Skipped 2 previous similar messages [ 3720.672918] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3724.017645] Lustre: server umount lustre-OST0000 complete [ 3731.648603] Lustre: server umount lustre-MDT0000 complete [ 3733.130266] LustreError: 135524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781612627 with bad export cookie 13039444663509270960 [ 3733.133692] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3733.135589] LustreError: 135524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3733.271884] Lustre: server umount lustre-MDT0001 complete [ 3737.509349] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 3738.682191] Key type lgssc unregistered [ 3738.825485] LNet: 146510:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3738.828971] LNetError: 146510:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3738.836419] LNet: Removed LNI 192.168.203.139@tcp [ 3739.183190] Key type .llcrypt unregistered [ 3739.185193] Key type ._llcrypt unregistered [ 3746.130588] Key type ._llcrypt registered [ 3746.132060] Key type .llcrypt registered [ 3746.170667] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 3752.873216] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 3753.203294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3753.211309] alg: No test for adler32 (adler32-zlib) [ 3754.093903] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 3754.189875] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 3755.783696] Key type lgssc registered [ 3756.237119] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3770.449383] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3770.456161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3771.555446] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3771.569598] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3771.607866] Lustre: lustre-MDT0000: new disk, initializing [ 3771.635474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3771.644246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3773.227538] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3777.816816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3777.864103] Lustre: 149682:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3777.898460] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3777.901303] Lustre: Skipped 1 previous similar message [ 3777.941482] Lustre: lustre-MDT0001: new disk, initializing [ 3777.980726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3777.997593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3778.001965] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3779.578920] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3783.660934] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3785.730649] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3788.472459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3788.587403] Lustre: lustre-OST0000: new disk, initializing [ 3788.589542] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3788.593835] Lustre: 151650:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3788.631201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3790.898851] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3795.186720] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3796.488628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3796.494504] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3796.550499] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3804.145647] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_module obdecho/obdecho [ 3804.834223] LustreError: 152654:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 3804.839769] LustreError: 152654:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 3805.514994] LustreError: 152750:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 3811.809245] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3811.810664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3811.814170] Lustre: Skipped 2 previous similar messages [ 3811.817520] Lustre: Skipped 1 previous similar message [ 3815.131074] Lustre: server umount lustre-MDT0000 complete [ 3816.540334] LustreError: 149675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781612710 with bad export cookie 10735890167162123249 [ 3816.542684] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3816.545493] LustreError: 149675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3816.668719] Lustre: server umount lustre-MDT0001 complete [ 3828.389837] Lustre: server umount lustre-OST0000 complete [ 3836.397591] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 3840.327703] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 3853.718172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3853.807745] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3853.821432] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3853.858215] Lustre: lustre-MDT0000: new disk, initializing [ 3853.884984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3853.891949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3855.288529] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3859.691961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3859.737338] Lustre: 156458:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3859.761371] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3859.763554] Lustre: Skipped 1 previous similar message [ 3859.800784] Lustre: lustre-MDT0001: new disk, initializing [ 3859.842510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3859.858167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3859.862614] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3861.342454] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3865.260735] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3867.341795] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3869.868048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3869.986394] Lustre: lustre-OST0000: new disk, initializing [ 3869.989153] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3869.993895] Lustre: 158429:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3870.031505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3872.116048] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3875.991956] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3877.513705] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3877.894617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3877.901420] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3877.933185] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3878.616661] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3880.085754] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3880.164809] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3882.978797] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3882.981605] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3882.982263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3882.984836] Lustre: Skipped 1 previous similar message [ 3886.930581] Lustre: server umount lustre-OST0000 complete [ 3890.655819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3890.658970] LustreError: Skipped 1 previous similar message [ 3890.660959] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3890.665964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3890.668018] Lustre: Skipped 1 previous similar message [ 3893.216738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3893.219916] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3894.363351] Lustre: server umount lustre-MDT0000 complete [ 3895.650140] LustreError: 156451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781612790 with bad export cookie 10735890167162124201 [ 3895.652485] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3895.654381] LustreError: 156451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3895.774496] Lustre: server umount lustre-MDT0001 complete [ 3896.907843] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 08:26:30 (1781612790) [ 3897.424846] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 3898.016634] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 08:26:31 (1781612791) [ 3900.639415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3900.808045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3902.165846] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3905.097442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3906.671637] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3909.294276] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3911.105366] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3913.439299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3915.571739] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3918.250984] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3931.434599] LustreError: 163438:0:(ldlm_resource.c:1180:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x200000bd1:0x2f:0x0].0x0 (00000000f92c5919) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3932.129769] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3932.131250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3932.134411] Lustre: Skipped 2 previous similar messages [ 3932.137173] Lustre: Skipped 2 previous similar messages [ 3937.575113] Lustre: server umount lustre-MDT0000 complete [ 3938.942340] LustreError: 160822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781612833 with bad export cookie 10735890167162125069 [ 3938.945945] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3938.947858] LustreError: 160822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3939.122917] LustreError: 163027:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3939.131743] LustreError: 163027:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3942.367908] LustreError: 160835:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3942.368222] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3942.380536] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3942.383013] Lustre: Skipped 6 previous similar messages [ 3943.391811] LustreError: 162272:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3945.160213] Lustre: server umount lustre-MDT0001 complete [ 3947.563219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3952.156694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3952.263231] LustreError: 164698:0:(llog.c:1646:llog_backup()) MGC192.168.203.139@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 3952.266702] Lustre: 164698:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.139@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 3967.904966] LustreError: 164703:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3967.911714] LustreError: 164703:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3973.024958] LustreError: 164703:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3973.030793] LustreError: 164703:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3981.109091] LustreError: 164703:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3981.117922] LustreError: 164703:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 3998.176075] LustreError: 164721:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3998.185259] LustreError: 164721:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 4013.151440] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 4013.153913] Lustre: 164698:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.203.139@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 4031.282830] LustreError: 164703:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4031.291020] LustreError: 164703:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 20 previous similar messages [ 4053.994512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4053.999184] Lustre: Skipped 2 previous similar messages [ 4055.547308] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4058.798585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4059.108603] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4059.113226] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x94fd8b39d14ed435 [ 4060.680377] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4063.415500] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4064.031921] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4064.231351] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4064.234435] Lustre: Skipped 1 previous similar message [ 4066.067530] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4067.658933] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 1 sec [ 4074.466442] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4074.466578] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4074.467129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4074.467133] Lustre: Skipped 2 previous similar messages [ 4074.472928] Lustre: Skipped 1 previous similar message [ 4079.254186] Lustre: server umount lustre-MDT0000 complete [ 4080.503977] LustreError: 160820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781612974 with bad export cookie 10735890167162131509 [ 4080.507230] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4080.508924] LustreError: 160820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4080.621889] Lustre: server umount lustre-MDT0001 complete [ 4092.197052] Lustre: server umount lustre-OST0000 complete [ 4099.578599] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 4102.968772] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 4116.405412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4116.507021] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4116.520019] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4116.556735] Lustre: lustre-MDT0000: new disk, initializing [ 4116.588217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4118.116091] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4122.711286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4122.752508] Lustre: 169772:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4122.847725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4122.849864] Lustre: Skipped 2 previous similar messages [ 4122.866447] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4124.423142] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4128.444299] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4130.650566] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4133.421777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4133.533713] Lustre: 171742:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4135.927446] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4140.038360] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4140.548312] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4140.599093] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4141.690649] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4141.775786] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4143.388543] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4143.473570] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4145.636477] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4145.636911] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4145.637797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4145.637802] Lustre: Skipped 2 previous similar messages [ 4145.641712] Lustre: Skipped 2 previous similar messages [ 4145.651334] LustreError: Skipped 1 previous similar message [ 4146.265216] Lustre: server umount lustre-OST0000 complete [ 4153.864750] Lustre: server umount lustre-MDT0000 complete [ 4155.422360] LustreError: 169764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781613049 with bad export cookie 10735890167162148309 [ 4155.424654] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4155.427417] LustreError: 169764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4155.547320] Lustre: server umount lustre-MDT0001 complete [ 4158.810839] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 08:30:52 (1781613052) [ 4161.255212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4161.457097] LustreError: 174151:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4161.464617] LustreError: 174151:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 22 previous similar messages [ 4162.997281] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4166.392197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4168.156689] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4171.044300] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4173.116889] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4175.724294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4178.101233] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4180.894740] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4190.303596] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 08:31:23 (1781613083) [ 4196.321853] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4196.325871] LustreError: Skipped 2 previous similar messages [ 4197.457391] Lustre: server umount lustre-OST0000 complete [ 4205.169248] Lustre: server umount lustre-MDT0000 complete [ 4207.863186] LustreError: 174131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781613102 with bad export cookie 10735890167162149177 [ 4207.870830] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4207.875347] LustreError: 174131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4208.068546] Lustre: server umount lustre-MDT0001 complete [ 4223.063032] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 4226.478400] Key type lgssc unregistered [ 4226.739443] LNet: 177875:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4226.746729] LNetError: 177875:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4226.770039] LNet: Removed LNI 192.168.203.139@tcp [ 4227.870975] Key type .llcrypt unregistered [ 4227.876446] Key type ._llcrypt unregistered [ 4251.119377] Key type ._llcrypt registered [ 4251.121575] Key type .llcrypt registered [ 4251.202836] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 4252.277258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4252.347814] alg: No test for adler32 (adler32-zlib) [ 4253.857864] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 4254.111859] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 4255.855224] Key type lgssc registered [ 4257.823973] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4265.713811] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4265.746940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4281.311372] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4286.431217] LustreError: 179118:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.139@tcp: failed processing log, type 4: rc = -110 [ 4291.551682] LustreError: 179118:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4291.565526] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 4291.568645] LustreError: MGC192.168.203.139@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 4291.575225] LustreError: 179118:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-OST0000: -22 [ 4291.580654] LustreError: 179118:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -22 [ 4291.586751] LustreError: 179118:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 4291.635348] Lustre: server umount lustre-OST0000 complete [ 4291.637845] LustreError: 179118:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4297.657345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4298.950545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4301.732025] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4307.350604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4307.568445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4310.312932] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4314.856409] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4318.309057] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4322.745834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4322.898704] LustreError: 181430:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4322.906291] Lustre: 181430:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.139@tcp: invalid local config log lustre-OST0000: rc = -22 [ 4322.983597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4327.129136] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4331.716442] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4337.634566] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4337.636399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4337.640744] Lustre: Skipped 1 previous similar message [ 4339.405749] Lustre: server umount lustre-MDT0000 complete [ 4341.627356] LustreError: 179597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781613236 with bad export cookie 14199032686097002108 [ 4341.629230] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4341.635215] LustreError: 179597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4341.822336] Lustre: server umount lustre-MDT0001 complete [ 4354.425713] Lustre: server umount lustre-OST0000 complete [ 4360.641131] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 4366.902357] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 4389.316378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4389.526993] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4389.563589] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4389.630583] Lustre: lustre-MDT0000: new disk, initializing [ 4389.716894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4389.729956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4393.001486] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4400.103064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4400.182435] Lustre: 184955:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4400.226971] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4400.231260] Lustre: Skipped 1 previous similar message [ 4400.285994] Lustre: lustre-MDT0001: new disk, initializing [ 4400.337690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4400.360529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4400.371359] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4402.913157] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4409.515212] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4412.958663] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4417.350981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4417.508299] Lustre: lustre-OST0000: new disk, initializing [ 4417.511297] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4417.515402] Lustre: 186930:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4417.572901] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4421.390109] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4425.782147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4425.788537] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4425.874597] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4427.162348] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4430.430124] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4430.647219] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4434.251624] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4434.419106] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4436.449438] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4436.455037] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4436.457399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4436.470010] Lustre: Skipped 2 previous similar messages [ 4436.475461] Lustre: Skipped 2 previous similar messages [ 4441.571544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4441.577793] Lustre: Skipped 1 previous similar message [ 4441.844695] Lustre: server umount lustre-OST0000 complete [ 4446.689068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4446.689661] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4446.714168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4446.715235] LustreError: Skipped 1 previous similar message [ 4446.728458] Lustre: Skipped 1 previous similar message [ 4450.666894] Lustre: server umount lustre-MDT0000 complete [ 4451.813677] LustreError: 184960:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4451.840486] LustreError: 184960:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4453.104841] LustreError: 184946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781613347 with bad export cookie 14199032686097002829 [ 4453.108674] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4453.113347] LustreError: 184946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4453.326509] Lustre: server umount lustre-MDT0001 complete [ 4458.821902] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 08:35:51 (1781613351) [ 4470.090203] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 4471.753965] Key type lgssc unregistered [ 4471.935756] LNet: 189873:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4471.942566] LNetError: 189873:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4471.963381] LNet: Removed LNI 192.168.203.139@tcp [ 4472.524309] Key type .llcrypt unregistered [ 4472.525811] Key type ._llcrypt unregistered [ 4483.901300] Key type ._llcrypt registered [ 4483.904267] Key type .llcrypt registered [ 4483.966698] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 4484.508780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4484.587051] alg: No test for adler32 (adler32-zlib) [ 4485.550726] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 4485.723806] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 4487.367176] Key type lgssc registered [ 4488.202623] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4493.613965] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4493.635408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4494.960371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4498.275838] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4504.434010] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4504.717919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4507.760929] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4512.413122] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4516.292932] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4521.655525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4521.811677] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4521.816612] LustreError: 192930:0:(llog.c:1687:llog_backup()) MGC192.168.203.139@tcp: failed to backup log lustre-OST0000: rc = -28 [ 4521.825639] Lustre: 192930:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.139@tcp: can't backup local config lustre-OST0000: rc = -28 [ 4521.905819] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4526.191384] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4531.148466] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4535.267248] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4535.268323] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4535.294879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4535.310871] Lustre: Skipped 2 previous similar messages [ 4538.897894] Lustre: server umount lustre-MDT0000 complete [ 4540.385674] LustreError: 191063:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4540.402705] LustreError: 191063:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4541.319340] LustreError: 191051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781613435 with bad export cookie 15458246912915144194 [ 4541.326951] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4541.334530] LustreError: 191051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4541.512155] Lustre: server umount lustre-MDT0001 complete [ 4554.387965] Lustre: server umount lustre-OST0000 complete [ 4560.279995] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 4566.464538] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 4588.729896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4588.909691] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4588.942794] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4589.015914] Lustre: lustre-MDT0000: new disk, initializing [ 4589.079161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4589.091315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4591.432297] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4597.626717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4597.693531] Lustre: 196449:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4597.727845] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4597.731220] Lustre: Skipped 1 previous similar message [ 4597.733414] LustreError: 197233:0:(llog.c:1687:llog_backup()) MGC192.168.203.139@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 4597.737347] LustreError: 197233:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 4597.741374] Lustre: 197233:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.139@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 4597.753516] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4597.760073] Lustre: Skipped 1 previous similar message [ 4597.820890] Lustre: lustre-MDT0001: new disk, initializing [ 4597.886784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4597.906888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4597.915706] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4600.272846] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4606.248636] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4609.362852] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4613.368431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4613.509501] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4613.511499] Lustre: Skipped 1 previous similar message [ 4613.513946] LustreError: 198395:0:(llog.c:1687:llog_backup()) MGC192.168.203.139@tcp: failed to backup log lustre-OST0000: rc = -28 [ 4613.519194] LustreError: 198395:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 4613.523223] Lustre: 198395:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.139@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 4613.531137] Lustre: 198395:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 4613.548334] Lustre: lustre-OST0000: new disk, initializing [ 4613.550502] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4613.553982] Lustre: 198419:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4613.594933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4616.829825] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4622.341751] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4622.860251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4622.866313] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4622.928469] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4624.669558] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4624.805547] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4627.280599] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4627.442845] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4630.722558] Lustre: server umount lustre-OST0000 complete [ 4633.057857] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4633.063658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4633.068036] Lustre: Skipped 3 previous similar messages [ 4638.177106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4638.183210] Lustre: Skipped 2 previous similar messages [ 4643.297344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4643.301703] Lustre: Skipped 1 previous similar message [ 4647.391931] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4647.565651] Lustre: server umount lustre-MDT0000 complete [ 4648.417215] LustreError: 197578:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4648.431686] LustreError: 197578:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4649.312090] Lustre: 190383:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781613527/real 1781613527] req@00000000ef2a0db7 x1868157031603072/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1781613543 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4649.333563] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4649.980527] LustreError: 197247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781613544 with bad export cookie 15458246912915144908 [ 4649.984639] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4649.988578] LustreError: 197247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4650.240976] Lustre: server umount lustre-MDT0001 complete [ 4656.610206] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 08:39:09 (1781613549) [ 4666.587972] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 4673.391249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4673.629223] LustreError: 201544:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4673.642553] LustreError: 201544:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4673.656324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4676.606714] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4679.136847] LustreError: 201545:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4682.285851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4685.431650] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4687.334915] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4691.409887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4695.603525] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4702.243023] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4702.388984] Lustre: lustre-OST0001: new disk, initializing [ 4702.391683] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4702.396660] Lustre: 203852:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4706.775671] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4709.446157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4709.458575] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4709.535414] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4714.874829] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4720.401643] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4721.633709] Lustre: DEBUG MARKER: ADD OST3 [ 4724.996793] loop: module loaded [ 4725.028778] loop0: detected capacity change from 0 to 204800000 [ 4725.162975] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4725.174560] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4725.183396] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4729.341824] loop0: detected capacity change from 0 to 204800000 [ 4731.107144] Lustre: DEBUG MARKER: STOP OST3 [ 4733.188834] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4733.340137] Lustre: lustre-OST0003: new disk, initializing [ 4733.342814] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 4733.347250] Lustre: 205784:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4733.428828] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 4733.434140] Lustre: Skipped 3 previous similar messages [ 4734.769935] Lustre: Failing over lustre-OST0003 [ 4734.879795] Lustre: server umount lustre-OST0003 complete [ 4735.370764] LustreError: 204346:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4737.449796] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4744.733588] Lustre: DEBUG MARKER: START OST3 again [ 4745.016452] LustreError: 203029:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4745.027121] LustreError: 203029:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4749.161705] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4749.397730] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 4750.136761] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 1 client reconnects [ 4750.157272] Lustre: lustre-OST0003: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 4752.746293] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4754.418061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 4754.424513] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 4754.458087] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 4768.565672] Lustre: Failing over lustre-OST0003 [ 4768.624871] Lustre: server umount lustre-OST0003 complete [ 4769.761371] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4769.765619] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4769.766561] LustreError: Skipped 1 previous similar message [ 4769.781282] LustreError: 203032:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4769.786496] Lustre: Skipped 1 previous similar message [ 4769.808642] LustreError: 203032:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4785.121904] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4785.123020] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4785.126621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4785.146475] Lustre: Skipped 3 previous similar messages [ 4795.364849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4795.371275] Lustre: Skipped 10 previous similar messages [ 4797.407185] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4797.558473] Lustre: server umount lustre-MDT0000 complete [ 4799.758929] LustreError: 201524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781613694 with bad export cookie 15458246912915145790 [ 4799.762415] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4799.763753] LustreError: 201524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4800.482048] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4800.490691] Lustre: Skipped 1 previous similar message [ 4804.514301] LustreError: 202889:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4804.523343] LustreError: 202889:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 4814.305437] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4814.456161] Lustre: server umount lustre-MDT0001 complete [ 4816.684596] Lustre: server umount lustre-OST0000 complete [ 4819.055884] Lustre: server umount lustre-OST0001 complete [ 4823.057442] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 4827.941687] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 4850.425809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4850.554411] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4850.578558] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4850.632388] Lustre: lustre-MDT0000: new disk, initializing [ 4850.685793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4850.688647] Lustre: Skipped 1 previous similar message [ 4850.706032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4852.909515] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4859.560804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4859.626467] Lustre: 211304:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4859.795201] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4862.328395] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4868.281603] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4871.249036] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4874.760205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4874.915471] Lustre: lustre-OST0000: new disk, initializing [ 4874.918360] Lustre: Skipped 1 previous similar message [ 4874.920964] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4874.923647] Lustre: Skipped 2 previous similar messages [ 4874.927365] Lustre: 213270:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4876.299089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4876.304670] Lustre: Skipped 1 previous similar message [ 4876.310678] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4876.360632] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4878.003394] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4883.570332] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4886.278055] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4886.421679] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4888.861797] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4889.017734] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4891.617693] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4891.618543] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4891.627143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4891.630372] LustreError: Skipped 1 previous similar message [ 4891.634348] Lustre: Skipped 6 previous similar messages [ 4896.323688] Lustre: server umount lustre-OST0000 complete [ 4900.832469] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4905.085150] Lustre: server umount lustre-MDT0000 complete [ 4906.976659] LustreError: 211316:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4906.989325] LustreError: 211316:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4907.448623] LustreError: 211297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781613801 with bad export cookie 15458246912915148072 [ 4907.453391] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4907.457395] LustreError: 211297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4907.667807] Lustre: server umount lustre-MDT0001 complete [ 4909.959106] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 08:43:22 (1781613802) [ 4919.328092] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 4924.860987] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 4946.923950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4947.117604] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4947.154472] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4947.215448] Lustre: lustre-MDT0000: new disk, initializing [ 4947.268839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4949.658440] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4956.334518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4956.431810] Lustre: 217874:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4956.605501] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4959.454273] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4965.483215] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4968.798047] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4972.686188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4972.858372] Lustre: 219839:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4974.956853] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4976.532691] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 4982.124247] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4984.864286] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4985.011531] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4987.638312] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4987.768914] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4990.436583] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4990.437783] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4990.443921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4990.455814] Lustre: Skipped 4 previous similar messages [ 4990.462262] Lustre: Skipped 4 previous similar messages [ 5003.231147] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5003.311404] Lustre: server umount lustre-OST0000 complete [ 5011.736995] Lustre: server umount lustre-MDT0000 complete [ 5013.885271] LustreError: 218668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781613908 with bad export cookie 15458246912915148954 [ 5013.888024] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5013.894718] LustreError: 218668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5014.060202] Lustre: server umount lustre-MDT0001 complete [ 5018.652243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5018.959642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5018.963211] Lustre: Skipped 5 previous similar messages [ 5021.372900] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5026.058064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5028.793559] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5032.894735] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5036.099793] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5040.305816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5044.591608] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5049.037525] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5059.040676] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5059.046661] LustreError: Skipped 1 previous similar message [ 5059.049797] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5059.066549] Lustre: Skipped 2 previous similar messages [ 5059.071904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5059.074977] Lustre: Skipped 9 previous similar messages [ 5063.579407] Lustre: server umount lustre-OST0000 complete [ 5072.213727] Lustre: server umount lustre-MDT0000 complete [ 5074.535063] LustreError: 222036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781613968 with bad export cookie 15458246912915149836 [ 5074.536556] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5074.544454] LustreError: 222036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5074.726784] Lustre: server umount lustre-MDT0001 complete [ 5185.415509] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 5187.322840] Key type lgssc unregistered [ 5187.523809] LNet: 225717:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5187.531685] LNetError: 225717:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5187.546494] LNet: Removed LNI 192.168.203.139@tcp [ 5188.093144] Key type .llcrypt unregistered [ 5188.094959] Key type ._llcrypt unregistered [ 5204.271967] Key type ._llcrypt registered [ 5204.274032] Key type .llcrypt registered [ 5204.339801] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 5214.741859] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 5215.348721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5215.379044] alg: No test for adler32 (adler32-zlib) [ 5216.307596] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 5216.474614] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 5218.095184] Key type lgssc registered [ 5218.812264] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5240.704711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5240.718382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5241.884620] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5241.920617] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5241.969336] Lustre: lustre-MDT0000: new disk, initializing [ 5242.035363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5242.055692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5244.593832] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5251.053111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5251.124714] Lustre: 229041:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5251.168682] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5251.172140] Lustre: Skipped 1 previous similar message [ 5251.217640] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5251.233696] Lustre: lustre-MDT0001: new disk, initializing [ 5251.300568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5251.328944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5251.339249] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5254.017633] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5259.716465] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5262.741326] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5266.712786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5266.858373] Lustre: lustre-OST0000: new disk, initializing [ 5266.860970] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5266.865086] Lustre: 231011:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5266.923983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5269.465763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5269.470978] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5269.557671] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5270.507155] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5276.449591] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5279.188227] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5279.329195] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5282.206225] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5282.354632] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5284.833405] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5284.839305] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5284.854196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5284.860113] Lustre: Skipped 1 previous similar message [ 5289.602755] Lustre: server umount lustre-OST0000 complete [ 5292.514123] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5292.522767] LustreError: Skipped 1 previous similar message [ 5292.528484] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5292.543575] Lustre: Skipped 1 previous similar message [ 5292.550373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5295.072921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5295.076329] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5300.193062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5300.196463] Lustre: Skipped 2 previous similar messages [ 5306.335156] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5306.421150] Lustre: server umount lustre-MDT0000 complete [ 5308.655282] LustreError: 231009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781614203 with bad export cookie 7265600803743256106 [ 5308.660681] LustreError: 231009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5308.661187] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5308.830805] Lustre: server umount lustre-MDT0001 complete [ 5318.529492] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 5324.667602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5324.930151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5327.275266] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5330.401277] LustreError: 233943:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5330.408773] LustreError: 233943:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5331.592821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5331.797615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5334.043830] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5335.683545] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5339.209112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5342.414993] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5346.915201] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5347.022377] Lustre: lustre-OST0001: new disk, initializing [ 5347.027649] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5347.036559] Lustre: 236250:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5347.098355] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5347.102925] Lustre: Skipped 1 previous similar message [ 5348.142702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5348.151549] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5348.208847] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5350.213914] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5358.970852] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5361.638417] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5368.898139] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 08:51:01 (1781614261) [ 5370.849684] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5370.856442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5370.863730] Lustre: Skipped 3 previous similar messages [ 5370.873303] Lustre: Skipped 5 previous similar messages [ 5376.369418] Lustre: server umount lustre-MDT0000 complete [ 5378.583779] LustreError: 233925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781614272 with bad export cookie 7265600803743256981 [ 5378.584390] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5378.592112] LustreError: 233925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5378.791037] Lustre: server umount lustre-MDT0001 complete [ 5391.396397] Lustre: server umount lustre-OST0000 complete [ 5403.700670] Lustre: server umount lustre-OST0001 complete [ 5406.752296] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 5411.710112] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 5428.785833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5428.921806] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5428.940067] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5428.985889] Lustre: lustre-MDT0000: new disk, initializing [ 5429.039309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5429.061198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5431.290619] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5437.371637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5437.442965] Lustre: 240192:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5437.482431] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5437.484816] Lustre: Skipped 1 previous similar message [ 5437.527176] Lustre: lustre-MDT0001: new disk, initializing [ 5437.597720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5437.604429] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5439.541689] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5444.794688] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5447.544261] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5450.631366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5450.770617] Lustre: 242162:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5452.685116] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5452.753064] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5453.710148] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5458.673488] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5460.580544] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5460.682883] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5462.678547] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5462.776677] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5468.129168] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5468.131389] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5468.132909] LustreError: Skipped 1 previous similar message [ 5468.133745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5468.139946] Lustre: Skipped 1 previous similar message [ 5468.141693] Lustre: Skipped 4 previous similar messages [ 5469.800512] Lustre: server umount lustre-OST0000 complete [ 5477.870057] Lustre: server umount lustre-MDT0000 complete [ 5478.368268] LustreError: 240205:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5478.379458] LustreError: 240205:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5479.484295] LustreError: 240186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781614373 with bad export cookie 7265600803743259102 [ 5479.488562] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5479.491889] LustreError: 240186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5479.673398] Lustre: server umount lustre-MDT0001 complete [ 5483.803594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5484.037604] LustreError: 244373:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5484.084959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5484.089554] Lustre: Skipped 2 previous similar messages [ 5486.194165] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5489.121566] LustreError: 244374:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5490.430711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5492.734714] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5496.258729] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5498.916605] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5502.210295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5505.181387] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5508.588901] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5652.960876] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5652.961491] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5652.967907] LustreError: Skipped 1 previous similar message [ 5652.970089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5652.975458] Lustre: Skipped 3 previous similar messages [ 5652.978413] Lustre: Skipped 3 previous similar messages [ 5666.271126] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5666.357184] Lustre: server umount lustre-OST0000 complete [ 5669.856609] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5669.863387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5674.155110] Lustre: server umount lustre-MDT0000 complete [ 5675.954334] LustreError: 244355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781614570 with bad export cookie 7265600803743259984 [ 5675.956704] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5675.963408] LustreError: 244355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5676.164779] Lustre: server umount lustre-MDT0001 complete [ 5774.315677] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 5775.785231] Key type lgssc unregistered [ 5775.963812] LNet: 247983:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5775.967860] LNetError: 247983:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5775.977197] LNet: Removed LNI 192.168.203.139@tcp [ 5776.464237] Key type .llcrypt unregistered [ 5776.466792] Key type ._llcrypt unregistered [ 5789.640864] Key type ._llcrypt registered [ 5789.642696] Key type .llcrypt registered [ 5789.702235] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 5798.534603] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 5799.091157] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5799.116338] alg: No test for adler32 (adler32-zlib) [ 5800.033231] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 5800.158924] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 5801.775220] Key type lgssc registered [ 5802.438575] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5822.275262] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 5826.553463] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5826.561993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5827.680322] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5827.698138] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5827.740626] Lustre: lustre-MDT0000: new disk, initializing [ 5827.783840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5827.795424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5829.462355] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5834.501751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5834.557110] Lustre: 252024:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5834.595320] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5834.597433] Lustre: Skipped 1 previous similar message [ 5834.598146] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5834.636664] Lustre: lustre-MDT0001: new disk, initializing [ 5834.678646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5834.692782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5834.698770] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5836.508640] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5839.095736] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5841.705819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5841.833423] Lustre: lustre-OST0000: new disk, initializing [ 5841.835603] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5841.839602] Lustre: 253659:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5841.877477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5844.331037] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5849.413786] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5849.489420] Lustre: lustre-OST0001: new disk, initializing [ 5849.491313] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5849.494996] Lustre: 254521:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5849.540101] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5850.462369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5850.468483] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5850.548048] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5850.566719] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5852.272214] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5858.872062] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5862.158680] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5870.561872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5870.569681] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5870.582544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5871.082700] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5871.095580] Lustre: Skipped 2 previous similar messages [ 5873.694617] Lustre: server umount lustre-MDT0000 complete [ 5875.518589] LustreError: 252017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781614769 with bad export cookie 14500699704763976582 [ 5875.522741] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5875.526553] LustreError: 252017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5875.700926] Lustre: server umount lustre-MDT0001 complete [ 5887.953898] Lustre: server umount lustre-OST0000 complete [ 5898.866448] Lustre: server umount lustre-OST0001 complete [ 5905.743532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5910.244222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5913.908901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5918.258312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5926.678342] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 5931.201541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5931.213054] Lustre: lustre-MDT0000: reset Object Index mappings [ 5931.455218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5933.432496] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5936.609321] LustreError: 259528:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5936.618942] LustreError: 259528:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5937.386502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5937.399057] Lustre: lustre-MDT0001: reset Object Index mappings [ 5939.464750] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5940.838556] Lustre: 260668:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5944.023477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5944.223610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5944.228579] Lustre: Skipped 1 previous similar message [ 5947.304515] LustreError: 261022:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5947.366980] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5951.918576] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5955.447644] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5956.143950] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5956.146386] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 5959.908650] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5961.833611] Lustre: 262540:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5966.308274] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5966.313103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5966.318174] Lustre: Skipped 2 previous similar messages [ 5966.322442] Lustre: Skipped 5 previous similar messages [ 5970.075743] Lustre: server umount lustre-MDT0000 complete [ 5971.425922] LustreError: 259528:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5971.438515] LustreError: 259528:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5972.020174] LustreError: 260670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781614866 with bad export cookie 14500699704763983274 [ 5972.026342] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5972.036311] LustreError: 260670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5972.194515] Lustre: server umount lustre-MDT0001 complete [ 5983.782657] Lustre: server umount lustre-OST0000 complete [ 5996.130714] Lustre: server umount lustre-OST0001 complete [ 6006.038844] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 09:01:39 (1781614899) [ 6014.036834] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 6019.061966] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 6035.824505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6035.944619] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6035.962045] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6036.009514] Lustre: lustre-MDT0000: new disk, initializing [ 6036.056599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6036.060153] Lustre: Skipped 1 previous similar message [ 6036.068892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6036.073369] Lustre: Skipped 1 previous similar message [ 6037.929662] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6042.650684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6042.703820] Lustre: 266798:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6042.822101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6042.826570] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6042.831441] Lustre: Skipped 1 previous similar message [ 6044.434721] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6049.104312] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6051.596383] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6054.831710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6054.964548] Lustre: lustre-OST0000: new disk, initializing [ 6054.967087] Lustre: Skipped 1 previous similar message [ 6054.971481] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6054.974861] Lustre: Skipped 2 previous similar messages [ 6054.978371] Lustre: 268767:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6056.208148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6056.217267] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6056.261483] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6057.904887] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6062.806660] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6064.769341] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6064.864718] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6066.629550] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6066.726084] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6071.777336] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6071.778786] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6071.780550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6071.780556] Lustre: Skipped 2 previous similar messages [ 6071.784513] Lustre: Skipped 2 previous similar messages [ 6073.714274] Lustre: server umount lustre-OST0000 complete [ 6076.896665] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6076.897774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6076.904363] Lustre: Skipped 1 previous similar message [ 6076.906984] Lustre: Skipped 1 previous similar message [ 6081.729396] Lustre: server umount lustre-MDT0000 complete [ 6082.018561] LustreError: 266810:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6082.035770] LustreError: 266810:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6083.689292] LustreError: 266790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781614978 with bad export cookie 14500699704763985360 [ 6083.692560] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6083.699657] LustreError: 266790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6083.891614] Lustre: server umount lustre-MDT0001 complete [ 6094.060873] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 6099.472182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6099.702569] LustreError: 271699:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6099.732251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6099.734933] Lustre: Skipped 2 previous similar messages [ 6101.905170] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6106.344291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6108.670453] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6110.123773] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6113.090728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6115.498432] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6118.915975] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6119.007309] Lustre: lustre-OST0001: new disk, initializing [ 6119.009900] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6119.013460] Lustre: 274007:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6120.803261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6120.809794] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6120.847891] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6121.726673] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6127.071667] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6128.792134] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6131.223400] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 6131.231288] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 6136.291077] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6136.297491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6136.305140] Lustre: Skipped 3 previous similar messages [ 6136.308224] Lustre: Skipped 3 previous similar messages [ 6140.036246] Lustre: server umount lustre-MDT0000 complete [ 6141.411315] LustreError: 271700:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6141.422398] LustreError: 271700:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6141.783428] LustreError: 274006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781615036 with bad export cookie 14500699704763986242 [ 6141.786310] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6141.792088] LustreError: 274006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6141.962252] Lustre: server umount lustre-MDT0001 complete [ 6153.750395] Lustre: server umount lustre-OST0000 complete [ 6164.812548] Lustre: server umount lustre-OST0001 complete [ 6172.455113] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 6177.219182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6177.432978] LustreError: 277056:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6177.474655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6177.477458] Lustre: Skipped 3 previous similar messages [ 6179.498436] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6183.424656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6185.485318] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6186.739848] Lustre: 278197:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6189.409148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6192.090206] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6195.596151] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6197.898483] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6201.834835] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 6201.854632] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 6207.191059] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6208.813877] Lustre: 280074:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6211.199504] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 6211.201629] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 6215.371374] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 09:05:08 (1781615108) [ 6227.426197] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6227.427422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6227.431706] Lustre: Skipped 2 previous similar messages [ 6227.434116] Lustre: Skipped 1 previous similar message [ 6233.653883] Lustre: server umount lustre-MDT0000 complete [ 6235.510725] LustreError: 277037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781615129 with bad export cookie 14500699704763992542 [ 6235.511667] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6235.517876] LustreError: 277037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6237.664253] LustreError: 278993:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6237.670931] LustreError: 278993:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6241.757172] Lustre: server umount lustre-MDT0001 complete [ 6249.821124] Lustre: server umount lustre-OST0000 complete [ 6257.791328] Lustre: server umount lustre-OST0001 complete [ 6260.946139] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 6265.821361] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 6289.571259] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 6295.494588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6295.653844] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6295.676380] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6295.728381] Lustre: lustre-MDT0000: new disk, initializing [ 6295.790497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6298.444516] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6304.716605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6304.784561] Lustre: 284308:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6304.971634] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6307.216555] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6314.150574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6314.341090] Lustre: 285941:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6314.407875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6314.412526] Lustre: Skipped 5 previous similar messages [ 6316.290570] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6317.544752] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6323.269719] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6323.374595] Lustre: 286813:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6324.888118] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6324.892360] Lustre: Skipped 1 previous similar message [ 6324.936544] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6326.066336] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6330.760722] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6332.183877] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6332.186445] Lustre: Skipped 1 previous similar message [ 6337.346556] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 09:07:10 (1781615230) [ 6340.578483] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6340.581183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6340.584098] Lustre: Skipped 3 previous similar messages [ 6340.588845] Lustre: Skipped 14 previous similar messages [ 6344.424375] Lustre: server umount lustre-MDT0000 complete [ 6345.700316] LustreError: 284318:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6345.709765] LustreError: 284318:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6346.152123] LustreError: 284302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781615240 with bad export cookie 14500699704764000431 [ 6346.155240] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6346.158261] LustreError: 284302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6346.340585] Lustre: server umount lustre-MDT0001 complete [ 6358.023216] Lustre: server umount lustre-OST0000 complete [ 6368.758572] Lustre: server umount lustre-OST0001 complete [ 6372.508759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6374.670397] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6378.767711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6380.676841] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6383.671383] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6385.907251] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6388.782795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6391.428626] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6394.691764] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6407.071793] loop1: detected capacity change from 0 to 204800000 [ 6407.160896] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6407.166063] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6407.170777] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6409.271294] loop1: detected capacity change from 0 to 204800000 [ 6411.457361] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6411.496283] LustreError: 289043:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs15408: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 6411.501653] LustreError: 289043:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs15408 [ 6411.503889] LustreError: 289043:0:(mgs_handler.c:517:mgs_target_reg()) fs15408-MDT0001: Error checking target index: rc = -13 [ 6411.508532] LustreError: fs15408-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 6411.513369] LustreError: 292772:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [ 6411.516672] LustreError: 292772:0:(tgt_mount.c:2084:server_put_super()) no obd fs15408-MDT0001 [ 6411.519225] LustreError: 292772:0:(tgt_mount.c:116:server_deregister_mount()) fs15408-MDT0001 not registered [ 6411.531858] Lustre: server umount fs15408-MDT0001 complete [ 6411.534048] LustreError: 292772:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 6414.994508] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6415.032756] Lustre: Setting parameter fs15408-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15408-MDT0001 [ 6415.052274] Lustre: srv-fs15408-MDT0001: No data found on store. Initialize space. [ 6415.054166] Lustre: Skipped 4 previous similar messages [ 6415.091827] Lustre: fs15408-MDT0001: new disk, initializing [ 6415.093868] Lustre: Skipped 3 previous similar messages [ 6415.138195] Lustre: srv-fs15408-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 6415.142082] Lustre: 293236:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs15408-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6415.147719] LustreError: 293236:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs15408-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6416.786420] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6420.360436] Lustre: server umount fs15408-MDT0001 complete [ 6425.792624] loop1: detected capacity change from 0 to 204800000 [ 6428.093760] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6428.227655] Lustre: 294803:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs15408-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6428.231883] LustreError: 294803:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs15408-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6429.977306] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6432.170923] Lustre: server umount fs15408-MDT0001 complete [ 6439.393574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6439.398355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6439.401389] Lustre: Skipped 4 previous similar messages [ 6439.404168] Lustre: Skipped 1 previous similar message [ 6442.757237] Lustre: server umount lustre-MDT0000 complete [ 6446.506862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6446.571995] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6448.329220] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6451.551663] loop1: detected capacity change from 0 to 204800000 [ 6451.690631] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6453.999850] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6454.177555] Lustre: 297244:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs15408-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6454.183310] LustreError: 297244:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs15408-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6456.192971] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6458.510787] Lustre: server umount fs15408-MDT0001 complete [ 6472.171241] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6477.284296] LustreError: 291023:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6477.291688] LustreError: 291023:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 52 previous similar messages [ 6477.295720] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 6477.299223] LustreError: Skipped 1 previous similar message [ 6481.375099] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6481.451275] Lustre: server umount lustre-OST0000 complete [ 6487.521654] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6497.247086] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6497.320336] Lustre: server umount lustre-MDT0000 complete [ 6498.839600] LustreError: 289035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781615393 with bad export cookie 14500699704764003105 [ 6498.842729] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6498.845969] LustreError: 289035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6498.977197] Lustre: server umount lustre-MDT0001 complete [ 6503.061612] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 6504.142132] Key type lgssc unregistered [ 6504.267709] LNet: 299585:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6504.271835] LNetError: 299585:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6504.284572] LNet: Removed LNI 192.168.203.139@tcp [ 6504.617126] Key type .llcrypt unregistered [ 6504.618723] Key type ._llcrypt unregistered [ 6513.148190] Key type ._llcrypt registered [ 6513.149971] Key type .llcrypt registered [ 6513.197192] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 6513.671514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6513.678490] alg: No test for adler32 (adler32-zlib) [ 6514.544875] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 6514.640117] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 6516.231113] Key type lgssc registered [ 6516.626600] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6519.623053] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6519.631058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6520.850553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6522.470801] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6525.928071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6526.145127] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6527.827625] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6530.813920] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6533.168260] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6536.007526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6536.176808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6537.189860] LustreError: 302662:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6537.201971] LustreError: 302662:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6538.721880] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6541.800183] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6542.307587] LustreError: 302664:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6546.402161] LustreError: 302662:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6546.409992] LustreError: 302662:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6551.523876] LustreError: 302662:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6551.529751] LustreError: 302662:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6554.280960] loop1: detected capacity change from 0 to 204800000 [ 6554.411675] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6554.417333] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6554.425166] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6556.621755] loop1: detected capacity change from 0 to 204800000 [ 6556.642759] LustreError: 302662:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6556.650679] LustreError: 302662:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6559.208422] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6559.302319] Lustre: fs2967-OST0000: new disk, initializing [ 6559.307054] Lustre: srv-fs2967-OST0000: No data found on store. Initialize space. [ 6559.312259] Lustre: 304566:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6559.349570] Lustre: fs2967-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6562.279458] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6566.605645] Lustre: server umount fs2967-OST0000 complete [ 6566.881886] LustreError: 302806:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6566.889904] LustreError: 302806:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6571.551509] loop1: detected capacity change from 0 to 204800000 [ 6573.763565] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6573.848909] Lustre: fs2967-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6576.483575] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6580.639026] loop2: detected capacity change from 0 to 204800000 [ 6580.762052] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6580.774484] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6580.781919] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6583.096028] LustreError: 302662:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6583.106747] LustreError: 302662:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 6583.341101] loop2: detected capacity change from 0 to 204800000 [ 6585.665710] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6585.709220] LustreError: fs2967-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 6585.715985] LustreError: fs2967-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 6585.721601] LustreError: 307602:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [ 6585.725342] LustreError: 307602:0:(tgt_mount.c:2084:server_put_super()) no obd fs2967-OST0001 [ 6585.728227] LustreError: 307602:0:(tgt_mount.c:116:server_deregister_mount()) fs2967-OST0001 not registered [ 6585.743103] Lustre: server umount fs2967-OST0001 complete [ 6585.744971] LustreError: 307602:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 6590.451506] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6590.510290] Lustre: fs2967-OST0001: new disk, initializing [ 6590.513678] Lustre: srv-fs2967-OST0001: No data found on store. Initialize space. [ 6590.517888] Lustre: 308154:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6590.548934] Lustre: fs2967-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6593.383515] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6597.792247] Lustre: server umount fs2967-OST0000 complete [ 6602.315955] Lustre: server umount fs2967-OST0001 complete [ 6608.152910] loop1: detected capacity change from 0 to 204800000 [ 6610.727486] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6610.814305] Lustre: fs2967-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6613.593897] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6616.036542] LustreError: 302662:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6616.048624] LustreError: 302662:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 19 previous similar messages [ 6616.691215] loop2: detected capacity change from 0 to 204800000 [ 6619.313361] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6621.773462] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6625.719993] Lustre: server umount fs2967-OST0000 complete [ 6626.278971] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 6630.041352] Lustre: server umount fs2967-OST0001 complete [ 6631.397689] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [ 6636.514770] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6636.516347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6636.520740] Lustre: Skipped 1 previous similar message [ 6641.565807] Lustre: server umount lustre-OST0000 complete [ 6644.192061] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6644.196479] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6644.204497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6644.207989] Lustre: Skipped 1 previous similar message [ 6646.753197] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6646.753523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6646.761881] Lustre: Skipped 1 previous similar message [ 6649.443630] Lustre: server umount lustre-MDT0000 complete [ 6650.841635] LustreError: 300814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781615545 with bad export cookie 10283403732794767971 [ 6650.844858] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6650.847043] LustreError: 300814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6651.037409] Lustre: server umount lustre-MDT0001 complete [ 6655.153881] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 6656.350704] Key type lgssc unregistered [ 6656.488486] LNet: 314067:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6656.491830] LNetError: 314067:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6656.499597] LNet: Removed LNI 192.168.203.139@tcp [ 6656.889184] Key type .llcrypt unregistered [ 6656.891135] Key type ._llcrypt unregistered [ 6669.814678] Key type ._llcrypt registered [ 6669.816182] Key type .llcrypt registered [ 6669.859807] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 6676.824846] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 6677.139115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6677.150356] alg: No test for adler32 (adler32-zlib) [ 6678.014063] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 6678.109224] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 6679.703618] Key type lgssc registered [ 6680.115413] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6692.955395] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6692.961430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6694.059614] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6694.076586] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6694.116803] Lustre: lustre-MDT0000: new disk, initializing [ 6694.159179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6694.170405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6695.685265] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6700.283104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6700.331179] Lustre: 317898:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6700.362155] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6700.364056] Lustre: Skipped 1 previous similar message [ 6700.399949] Lustre: lustre-MDT0001: new disk, initializing [ 6700.437255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6700.454645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6700.458943] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6702.020620] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6705.913117] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6707.798404] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6710.401878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6710.518706] Lustre: lustre-OST0000: new disk, initializing [ 6710.522387] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6710.525876] Lustre: 319865:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6710.563535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6712.318081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6712.322109] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6712.358724] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6712.792864] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6716.810338] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6717.551524] Lustre: Mounted lustre-client [ 6717.912170] Lustre: Unmounted lustre-client [ 6722.528481] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6722.528676] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6722.532166] LustreError: Skipped 1 previous similar message [ 6722.539523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6722.543460] Lustre: Skipped 1 previous similar message [ 6723.044039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6723.045817] Lustre: Skipped 1 previous similar message [ 6727.119929] Lustre: server umount lustre-OST0000 complete [ 6731.231826] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6731.234348] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6731.238835] Lustre: Skipped 1 previous similar message [ 6731.241300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6733.280925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6733.281179] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6734.622276] Lustre: server umount lustre-MDT0000 complete [ 6735.859220] LustreError: 317888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781615630 with bad export cookie 4056664973368211542 [ 6735.862734] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6735.864553] LustreError: 317888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6735.967696] Lustre: server umount lustre-MDT0001 complete [ 6739.682810] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 6740.806935] Key type lgssc unregistered [ 6770.655581] LNet: 322023:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6770.658667] LNetError: 322023:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6850.533537] LNet: Removed LNI 192.168.203.139@tcp [ 6850.892805] Key type .llcrypt unregistered [ 6850.894513] Key type ._llcrypt unregistered [ 6857.848652] Key type ._llcrypt registered [ 6857.850519] Key type .llcrypt registered [ 6857.892829] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 6864.445354] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 6864.941860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6864.991919] alg: No test for adler32 (adler32-zlib) [ 6865.863885] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 6865.963414] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 6867.551186] Key type lgssc registered [ 6868.003668] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6879.617191] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 09:16:12 (1781615772) [ 6882.039209] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6882.046401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6883.156181] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6883.173333] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6883.215200] Lustre: lustre-MDT0000: new disk, initializing [ 6883.250333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6883.261863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6884.814337] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6889.434072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6889.481345] Lustre: 325221:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6889.509366] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6889.512170] Lustre: Skipped 1 previous similar message [ 6889.551318] Lustre: lustre-MDT0001: new disk, initializing [ 6889.591914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6889.608093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6889.613115] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6891.266418] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6895.498751] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6897.658196] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6900.166397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6900.309436] Lustre: lustre-OST0000: new disk, initializing [ 6900.311639] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6900.314514] Lustre: 327189:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6900.345789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6902.883173] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6907.080391] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6907.388329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6907.392265] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6907.435266] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6922.724713] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6922.725830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6922.726692] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6922.731959] Lustre: Skipped 1 previous similar message [ 6924.768745] Lustre: server umount lustre-OST0000 complete [ 6927.844053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6927.851512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6927.855334] Lustre: Skipped 2 previous similar messages [ 6932.443878] Lustre: server umount lustre-MDT0000 complete [ 6933.833392] LustreError: 325212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781615828 with bad export cookie 4870251665725367509 [ 6933.838327] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6933.838518] LustreError: 325212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6933.965930] Lustre: server umount lustre-MDT0001 complete [ 6938.039467] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 6939.161389] Key type lgssc unregistered [ 6939.292675] LNet: 329093:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6939.297066] LNetError: 329093:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6939.306561] LNet: Removed LNI 192.168.203.139@tcp [ 6939.637962] Key type .llcrypt unregistered [ 6939.639302] Key type ._llcrypt unregistered [ 6948.398146] Key type ._llcrypt registered [ 6948.399351] Key type .llcrypt registered [ 6948.443460] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 6948.854391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6948.874130] alg: No test for adler32 (adler32-zlib) [ 6949.755253] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 6949.871753] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 6951.471218] Key type lgssc registered [ 6951.911630] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6954.806269] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6954.820249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6956.021609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6957.497133] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6960.420094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6960.588556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6961.970957] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6964.226593] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6966.058700] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6968.434317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6968.555878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6970.629095] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6973.259887] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6987.468933] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6988.060530] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6991.331532] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6991.331887] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6991.332587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6991.335556] Lustre: Skipped 2 previous similar messages [ 6994.234791] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.39@tcp (stopping) [ 6994.237857] Lustre: Skipped 2 previous similar messages [ 6995.628949] Lustre: server umount lustre-MDT0000 complete [ 6996.448900] LustreError: 331083:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6996.454722] LustreError: 331083:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6996.924949] LustreError: 330327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781615891 with bad export cookie 10484073739355137160 [ 6996.927753] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6996.929972] LustreError: 330327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6999.347039] LustreError: 330342:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6999.359080] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.39@tcp (stopping) [ 7001.567840] LustreError: 330343:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7001.568464] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7001.581140] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7004.466815] LustreError: 332833:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7006.688213] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7006.690301] Lustre: Skipped 1 previous similar message [ 7009.586531] LustreError: 330343:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7009.594081] LustreError: 330343:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7011.295177] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7011.381709] Lustre: server umount lustre-MDT0001 complete [ 7018.880100] Lustre: server umount lustre-OST0000 complete [ 7022.391753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7023.966075] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7025.218981] Lustre: 334546:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.139@tcp [ 7026.140277] Lustre: server umount MGS complete [ 7029.304989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7029.468498] LustreError: 335118:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7029.491186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7030.894546] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7033.728584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7033.859595] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7035.178745] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7037.897146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7039.876022] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7042.310414] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7042.862875] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 7044.676928] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7045.223378] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7049.185560] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7049.186701] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7049.190904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7049.193163] LustreError: Skipped 1 previous similar message [ 7049.195266] Lustre: Skipped 2 previous similar messages [ 7052.434140] Lustre: server umount lustre-OST0000 complete [ 7054.305721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7054.305844] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7054.309820] Lustre: Skipped 1 previous similar message [ 7059.802620] Lustre: server umount lustre-MDT0000 complete [ 7061.199757] LustreError: 335100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781615955 with bad export cookie 10484073739355137986 [ 7061.201347] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7061.205867] LustreError: 335100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7061.331364] Lustre: server umount lustre-MDT0001 complete [ 7065.408740] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 7066.502181] Key type lgssc unregistered [ 7066.636571] LNet: 338546:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7066.640271] LNetError: 338546:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7066.654659] LNet: Removed LNI 192.168.203.139@tcp [ 7067.020560] Key type .llcrypt unregistered [ 7067.022260] Key type ._llcrypt unregistered [ 7073.800328] Key type ._llcrypt registered [ 7073.801699] Key type .llcrypt registered [ 7073.843966] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 7080.513544] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 7080.936637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7080.957364] alg: No test for adler32 (adler32-zlib) [ 7081.820559] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 7081.908404] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 7083.495158] Key type lgssc registered [ 7083.915403] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7095.819475] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7095.826838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7096.923691] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7096.938151] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7096.978457] Lustre: lustre-MDT0000: new disk, initializing [ 7097.008123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7097.019160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7098.599766] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7103.012109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7103.058381] Lustre: 341679:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7103.086400] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7103.088900] Lustre: Skipped 1 previous similar message [ 7103.129062] Lustre: lustre-MDT0001: new disk, initializing [ 7103.162252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7103.175718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7103.179704] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7104.694583] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7108.645133] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7110.571769] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7112.948306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7113.062664] Lustre: lustre-OST0000: new disk, initializing [ 7113.064374] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7113.067065] Lustre: 343648:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7113.098066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7114.041951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7114.045148] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7114.082714] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7115.149713] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7119.080819] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7120.580054] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7120.668793] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7122.155741] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7122.238566] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7124.449123] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7124.450514] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7124.451844] LustreError: Skipped 1 previous similar message [ 7124.455916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7124.458512] Lustre: Skipped 1 previous similar message [ 7129.570280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7129.572473] Lustre: Skipped 1 previous similar message [ 7134.689306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7134.691791] Lustre: Skipped 1 previous similar message [ 7137.247088] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7137.304318] Lustre: server umount lustre-OST0000 complete [ 7139.295909] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7139.299186] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7139.304424] Lustre: Skipped 1 previous similar message [ 7139.307117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7144.801439] Lustre: server umount lustre-MDT0000 complete [ 7146.279858] LustreError: 341672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781616040 with bad export cookie 12534504097863250803 [ 7146.283583] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7146.285498] LustreError: 341672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7146.414667] Lustre: server umount lustre-MDT0001 complete [ 7149.698598] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 09:20:43 (1781616043) [ 7155.459375] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 7158.703049] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 7171.315360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7171.411665] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7171.422425] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7171.457924] Lustre: lustre-MDT0000: new disk, initializing [ 7171.480168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7171.486816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7172.920327] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7177.184553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7177.230462] Lustre: 348397:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7177.261156] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7177.263465] Lustre: Skipped 1 previous similar message [ 7177.303158] Lustre: lustre-MDT0001: new disk, initializing [ 7177.343314] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7177.359121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7177.364102] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7178.812855] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7182.589541] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7184.492246] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7186.897266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7187.007900] Lustre: lustre-OST0000: new disk, initializing [ 7187.010221] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7187.014279] Lustre: 350365:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7187.050965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7187.214056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7187.218198] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7187.258285] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7188.971076] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7192.541518] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7202.784438] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7202.788646] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7202.790505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7202.793905] Lustre: Skipped 2 previous similar messages [ 7202.796350] Lustre: Skipped 1 previous similar message [ 7207.827288] Lustre: server umount lustre-OST0000 complete [ 7213.024914] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7213.025994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7213.028751] Lustre: Skipped 1 previous similar message [ 7213.030697] Lustre: Skipped 1 previous similar message [ 7215.190167] Lustre: server umount lustre-MDT0000 complete [ 7216.501085] LustreError: 349192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781616110 with bad export cookie 12534504097863251671 [ 7216.503798] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7216.506198] LustreError: 349192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7216.637749] Lustre: server umount lustre-MDT0001 complete [ 7220.236909] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 7221.352177] Key type lgssc unregistered [ 7221.493423] LNet: 352514:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7221.496221] LNetError: 352514:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7221.509389] LNet: Removed LNI 192.168.203.139@tcp [ 7221.829711] Key type .llcrypt unregistered [ 7221.831313] Key type ._llcrypt unregistered [ 7254.381811] Key type ._llcrypt registered [ 7254.383468] Key type .llcrypt registered [ 7254.432761] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 7254.556405] Key type .llcrypt unregistered [ 7254.557949] Key type ._llcrypt unregistered [ 7282.870427] Key type ._llcrypt registered [ 7282.871900] Key type .llcrypt registered [ 7282.913893] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 7283.019408] Key type .llcrypt unregistered [ 7283.020867] Key type ._llcrypt unregistered [ 7293.938870] Key type ._llcrypt registered [ 7293.940509] Key type .llcrypt registered [ 7293.976353] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 7294.082255] Key type .llcrypt unregistered [ 7294.083744] Key type ._llcrypt unregistered [ 7305.003709] Key type ._llcrypt registered [ 7305.004703] Key type .llcrypt registered [ 7305.038436] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 7305.135482] Key type .llcrypt unregistered [ 7305.137158] Key type ._llcrypt unregistered [ 7334.114909] Key type ._llcrypt registered [ 7334.116780] Key type .llcrypt registered [ 7334.163938] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 7334.279309] Key type .llcrypt unregistered [ 7334.281286] Key type ._llcrypt unregistered [ 7346.254121] Key type ._llcrypt registered [ 7346.255360] Key type .llcrypt registered [ 7346.289271] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 7346.393227] Key type .llcrypt unregistered [ 7346.394574] Key type ._llcrypt unregistered [ 7357.894912] Key type ._llcrypt registered [ 7357.896451] Key type .llcrypt registered [ 7357.934896] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 7358.043656] Key type .llcrypt unregistered [ 7358.045150] Key type ._llcrypt unregistered [ 7369.152660] Key type ._llcrypt registered [ 7369.154392] Key type .llcrypt registered [ 7369.199684] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 7369.308280] Key type .llcrypt unregistered [ 7369.309906] Key type ._llcrypt unregistered [ 7380.949698] Key type ._llcrypt registered [ 7380.950793] Key type .llcrypt registered [ 7380.990712] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 7381.094593] Key type .llcrypt unregistered [ 7381.095709] Key type ._llcrypt unregistered [ 7392.290319] Key type ._llcrypt registered [ 7392.291731] Key type .llcrypt registered [ 7392.326651] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 7392.427417] Key type .llcrypt unregistered [ 7392.428419] Key type ._llcrypt unregistered [ 7413.230455] Key type ._llcrypt registered [ 7413.231919] Key type .llcrypt registered [ 7413.271043] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 7413.385534] Key type .llcrypt unregistered [ 7413.386674] Key type ._llcrypt unregistered [ 7423.904653] Key type ._llcrypt registered [ 7423.905728] Key type .llcrypt registered [ 7423.942593] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 7424.036412] Key type .llcrypt unregistered [ 7424.038287] Key type ._llcrypt unregistered [ 7434.870369] Key type ._llcrypt registered [ 7434.871543] Key type .llcrypt registered [ 7434.915661] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 7435.032451] Key type .llcrypt unregistered [ 7435.033727] Key type ._llcrypt unregistered [ 7452.547783] Key type ._llcrypt registered [ 7452.549490] Key type .llcrypt registered [ 7452.589824] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 7458.375451] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 7458.714505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7458.769055] alg: No test for adler32 (adler32-zlib) [ 7459.616622] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 7459.693310] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 7461.263201] Key type lgssc registered [ 7461.588798] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7471.945091] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7471.953488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7473.039323] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 7473.050349] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 7473.085914] Lustre: tst250fs-MDT0000: new disk, initializing [ 7473.110322] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7473.116593] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7474.499211] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7478.650833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7478.689630] Lustre: 372417:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7478.726088] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 7478.729173] Lustre: Skipped 1 previous similar message [ 7478.765724] Lustre: tst250fs-MDT0001: new disk, initializing [ 7478.798880] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7478.812391] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7478.817066] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7480.193838] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7482.494673] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7484.735295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7484.843447] Lustre: tst250fs-OST0000: new disk, initializing [ 7484.845827] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 7484.849366] Lustre: 374050:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7484.880900] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7486.912933] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7486.916873] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7486.980319] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7487.072501] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7493.037716] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7493.153830] Lustre: tst250fs-OST0001: new disk, initializing [ 7493.158411] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 7493.163738] Lustre: 374923:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7493.249202] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7496.389773] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7496.395309] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7496.445626] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7496.794428] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7500.303590] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 7514.593328] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7514.600073] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7514.609935] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 7517.153109] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 7517.153347] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7520.007305] Lustre: server umount tst250fs-MDT0000 complete [ 7521.645942] LustreError: 373211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781616416 with bad export cookie 17285413554434660397 [ 7521.647223] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7521.651993] LustreError: 373211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7521.781064] Lustre: server umount tst250fs-MDT0001 complete [ 7533.444636] Lustre: server umount tst250fs-OST0000 complete [ 7545.124825] Lustre: server umount tst250fs-OST0001 complete [ 7552.540252] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 7555.751867] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 7567.455489] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 09:27:40 (1781616460) [ 7568.059434] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 7573.706977] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 7577.208571] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 7587.986128] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7418 sec ========== 09:28:01 (1781616481) [ 7588.596234] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 09:28:01 (1781616481) === [ 7589.776228] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 09:28:03 (1781616483) === [ 7604.063129] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 7605.161191] Key type lgssc unregistered [ 7605.302519] LNet: 384029:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7605.305966] LNetError: 384029:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7605.315343] LNet: Removed LNI 192.168.203.139@tcp [ 7605.639156] Key type .llcrypt unregistered [ 7605.640796] Key type ._llcrypt unregistered