[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 439534387 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002246] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.006767] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008009] pid_max: default: 32768 minimum: 301 [ 0.010049] LSM: Security Framework initializing [ 0.011078] Yama: becoming mindful. [ 0.012025] SELinux: Initializing. [ 0.013058] *** VALIDATE selinux *** [ 0.020113] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024743] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025168] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026086] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027085] *** VALIDATE tmpfs *** [ 0.028235] *** VALIDATE proc *** [ 0.029135] *** VALIDATE cgroup *** [ 0.030004] *** VALIDATE cgroup2 *** [ 0.030928] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.031138] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.032004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.033020] Spectre V2 : User space: Vulnerable [ 0.034004] Speculative Store Bypass: Vulnerable [ 0.036647] debug: unmapping init [mem 0xffffffff9c259000-0xffffffff9c260fff] [ 0.038129] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.039564] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.040014] ... version: 2 [ 0.040907] ... bit width: 48 [ 0.041009] ... generic registers: 4 [ 0.042005] ... value mask: 0000ffffffffffff [ 0.043007] ... max period: 00007fffffffffff [ 0.044006] ... fixed-purpose events: 3 [ 0.045007] ... event mask: 000000070000000f [ 0.046285] rcu: Hierarchical SRCU implementation. [ 0.048086] smp: Bringing up secondary CPUs ... [ 0.049438] x86: Booting SMP configuration: [ 0.050015] .... node #0, CPUs: #1 #2 #3 [ 0.052606] smp: Brought up 1 node, 4 CPUs [ 0.053858] smpboot: Max logical packages: 1 [ 0.054009] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.265985] node 0 deferred pages initialised in 211ms [ 0.269486] devtmpfs: initialized [ 0.270242] x86/mm: Memory block size: 128MB [ 0.272619] gcov: version magic: 0x41383552 [ 0.275312] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.276048] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.278417] pinctrl core: initialized pinctrl subsystem [ 0.280186] [ 0.280684] ************************************************************* [ 0.282009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.284006] ** ** [ 0.285007] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.287009] ** ** [ 0.289008] ** This means that this kernel is built to expose internal ** [ 0.290010] ** IOMMU data structures, which may compromise security on ** [ 0.292009] ** your system. ** [ 0.295012] ** ** [ 0.297009] ** If you see this message and you are not debugging the ** [ 0.299010] ** kernel, report this immediately to your vendor! ** [ 0.301012] ** ** [ 0.303009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.305009] ************************************************************* [ 0.306632] NET: Registered protocol family 16 [ 0.308467] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.312111] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.315055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.318610] cpuidle: using governor menu [ 0.320279] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.322553] PCI: Using configuration type 1 for base access [ 0.325146] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.334116] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.335021] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.336162] cryptd: max_cpu_qlen set to 1000 [ 0.339250] ACPI: Added _OSI(Module Device) [ 0.341012] ACPI: Added _OSI(Processor Device) [ 0.343014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.344011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.350632] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.356503] ACPI: Interpreter enabled [ 0.358056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.359007] ACPI: Using IOAPIC for interrupt routing [ 0.361145] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.364414] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.373226] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.376041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.377009] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.380067] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.383980] acpiphp: Slot [2] registered [ 0.385083] acpiphp: Slot [5] registered [ 0.386061] acpiphp: Slot [6] registered [ 0.387083] acpiphp: Slot [7] registered [ 0.388018] acpiphp: Slot [8] registered [ 0.389136] acpiphp: Slot [9] registered [ 0.391123] acpiphp: Slot [10] registered [ 0.392126] acpiphp: Slot [3] registered [ 0.394088] acpiphp: Slot [4] registered [ 0.395082] acpiphp: Slot [11] registered [ 0.396083] acpiphp: Slot [12] registered [ 0.398078] acpiphp: Slot [13] registered [ 0.399080] acpiphp: Slot [14] registered [ 0.400077] acpiphp: Slot [15] registered [ 0.402096] acpiphp: Slot [16] registered [ 0.403025] acpiphp: Slot [17] registered [ 0.404060] acpiphp: Slot [18] registered [ 0.405065] acpiphp: Slot [19] registered [ 0.406057] acpiphp: Slot [20] registered [ 0.407069] acpiphp: Slot [21] registered [ 0.408068] acpiphp: Slot [22] registered [ 0.409070] acpiphp: Slot [23] registered [ 0.411076] acpiphp: Slot [24] registered [ 0.412100] acpiphp: Slot [25] registered [ 0.414089] acpiphp: Slot [26] registered [ 0.415108] acpiphp: Slot [27] registered [ 0.417075] acpiphp: Slot [28] registered [ 0.418067] acpiphp: Slot [29] registered [ 0.419049] acpiphp: Slot [30] registered [ 0.420037] acpiphp: Slot [31] registered [ 0.421068] PCI host bridge to bus 0000:00 [ 0.422011] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.424028] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.426017] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.428020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.431023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.434024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.435157] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.436981] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.439845] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.449775] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.455470] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.458017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.460014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.462013] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.465478] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.468640] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.471043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.473767] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.480020] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.491023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.497021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.502768] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.509014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.514013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.531022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.539612] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.547014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.554016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.573015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.582264] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.587016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.592015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.608018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.615321] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.623014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.632018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.644018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.657822] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.661809] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.667012] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.680014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.687226] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.692882] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.697015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.716017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.728687] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.730336] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.733348] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.735310] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.738210] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.741216] iommu: Default domain type: Passthrough [ 0.743418] SCSI subsystem initialized [ 0.744133] ACPI: bus type USB registered [ 0.746077] usbcore: registered new interface driver usbfs [ 0.748087] usbcore: registered new interface driver hub [ 0.749077] usbcore: registered new device driver usb [ 0.751168] pps_core: LinuxPPS API ver. 1 registered [ 0.753013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.756074] PTP clock support registered [ 0.758180] EDAC MC: Ver: 3.0.0 [ 0.759426] PCI: Using ACPI for IRQ routing [ 0.761726] NetLabel: Initializing [ 0.762005] NetLabel: domain hash size = 128 [ 0.763010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.765123] NetLabel: unlabeled traffic allowed by default [ 0.767146] vgaarb: loaded [ 0.768267] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.770011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.775386] clocksource: Switched to clocksource kvm-clock [ 0.875761] VFS: Disk quotas dquot_6.6.0 [ 0.876957] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.879330] *** VALIDATE ramfs *** [ 0.880421] *** VALIDATE hugetlbfs *** [ 0.882455] pnp: PnP ACPI init [ 0.884777] pnp: PnP ACPI: found 6 devices [ 0.898747] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.901564] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.904228] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.906786] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.909628] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.911969] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.915066] NET: Registered protocol family 2 [ 0.917808] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.923104] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.926090] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.931092] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.934189] TCP: Hash tables configured (established 65536 bind 65536) [ 0.937504] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.941045] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.944158] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.947611] NET: Registered protocol family 1 [ 0.950692] RPC: Registered named UNIX socket transport module. [ 0.953421] RPC: Registered udp transport module. [ 0.955223] RPC: Registered tcp transport module. [ 0.956717] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.959229] NET: Registered protocol family 44 [ 0.960484] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.962750] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.965197] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.967484] PCI: CLS 0 bytes, default 64 [ 0.969569] Unpacking initramfs... [ 2.280381] debug: unmapping init [mem 0xffff9b5b3cc54000-0xffff9b5b3ffbffff] [ 2.284930] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.287462] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.289892] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.771671] Initialise system trusted keyrings [ 2.773640] Key type blacklist registered [ 2.779786] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.792309] zbud: loaded [ 2.795287] *** VALIDATE nfs *** [ 2.797406] *** VALIDATE nfs4 *** [ 2.799758] pstore: using deflate compression [ 2.803885] Platform Keyring initialized [ 2.911489] NET: Registered protocol family 38 [ 2.914414] Key type asymmetric registered [ 2.916351] Asymmetric key parser 'x509' registered [ 2.918894] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.922770] io scheduler mq-deadline registered [ 2.924731] io scheduler kyber registered [ 2.926784] io scheduler bfq registered [ 2.930187] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.933237] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.936803] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.940700] ACPI: Power Button [PWRF] [ 2.948521] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.957171] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.974784] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.983806] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.004227] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.031011] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.062024] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.067379] Non-volatile memory driver v1.3 [ 3.069641] Linux agpgart interface v0.103 [ 3.101502] virtio_blk virtio1: [vda] 145168 512-byte logical blocks (74.3 MB/70.9 MiB) [ 3.106116] vda: detected capacity change from 0 to 74326016 [ 3.122158] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.125821] vdb: detected capacity change from 0 to 1073741824 [ 3.140892] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.144582] vdc: detected capacity change from 0 to 2621440000 [ 3.163718] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.166903] vdd: detected capacity change from 0 to 2621440000 [ 3.185565] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.188711] vde: detected capacity change from 0 to 4294967296 [ 3.205718] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.208785] vdf: detected capacity change from 0 to 4294967296 [ 3.216075] libphy: Fixed MDIO Bus: probed [ 3.222115] usbcore: registered new interface driver usbserial_generic [ 3.224614] usbserial: USB Serial support registered for generic [ 3.227175] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.231848] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.233859] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.236316] mousedev: PS/2 mouse device common for all mice [ 3.239501] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.243438] rtc_cmos 00:05: RTC can wake from S4 [ 3.247659] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.250328] rtc_cmos 00:05: registered as rtc0 [ 3.250380] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.250405] intel_pstate: CPU model not supported [ 3.253299] hid: raw HID events driver (C) Jiri Kosina [ 3.257713] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.258282] usbcore: registered new interface driver usbhid [ 3.268154] usbhid: USB HID core driver [ 3.269902] drop_monitor: Initializing network drop monitor service [ 3.271975] Initializing XFRM netlink socket [ 3.273900] NET: Registered protocol family 10 [ 3.276606] Segment Routing with IPv6 [ 3.278067] NET: Registered protocol family 17 [ 3.280056] mpls_gso: MPLS GSO support [ 3.284743] RAS: Correctable Errors collector initialized. [ 3.286717] AVX version of gcm_enc/dec engaged. [ 3.288403] AES CTR mode by8 optimization enabled [ 3.370649] sched_clock: Marking stable (3370634723, 0)->(4101018323, -730383600) [ 3.373524] registered taskstats version 1 [ 3.375611] Loading compiled-in X.509 certificates [ 3.376793] zswap: loaded using pool lzo/zbud [ 3.396420] Key type big_key registered [ 3.406521] Key type encrypted registered [ 3.407693] ima: No TPM chip found, activating TPM-bypass! [ 3.409640] ima: Allocated hash algorithm: sha1 [ 3.410873] ima: No architecture policies found [ 3.411969] evm: Initialising EVM extended attributes: [ 3.413594] evm: security.selinux [ 3.414609] evm: security.ima [ 3.415215] evm: security.capability [ 3.416127] evm: HMAC attrs: 0x1 [ 3.418137] rtc_cmos 00:05: setting system clock to 2026-07-04 07:10:49 UTC (1783149049) [ 3.423125] debug: unmapping init [mem 0xffffffff9d203000-0xffffffff9d3fffff] [ 3.425973] debug: unmapping init [mem 0xffffffff9bf82000-0xffffffff9c258fff] [ 3.431344] Write protecting the kernel read-only data: 28672k [ 3.434220] debug: unmapping init [mem 0xffffffff9a603000-0xffffffff9a7fffff] [ 3.436183] debug: unmapping init [mem 0xffffffff9af14000-0xffffffff9affffff] [ 3.470372] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.478360] systemd[1]: Detected virtualization kvm. [ 3.479528] systemd[1]: Detected architecture x86-64. [ 3.481227] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.509391] systemd[1]: No hostname configured. [ 3.510918] systemd[1]: Set hostname to . [ 3.513013] random: systemd: uninitialized urandom read (16 bytes read) [ 3.515853] systemd[1]: Initializing machine ID from random generator. [ 3.635586] random: systemd: uninitialized urandom read (16 bytes read) [ 3.637680] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.645495] random: systemd: uninitialized urandom read (16 bytes read) [ 3.649355] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.654801] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Swap. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.301319] device-mapper: uevent: version 1.0.3 [ 4.303125] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK [0[ 4.863724] random: fast init done m] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.039052] virtio_net virtio0 ens2: renamed from eth0 [ 5.106510] scsi host0: ata_piix [ 5.139909] scsi host1: ata_piix [ 5.142071] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.144464] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.743498] dracut-initqueue[591]: RTNETLINK answers: File exists [ 9.920493] random: crng init done [ 9.922650] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.445961] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.702695] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.005444] SELinux: Disabled at runtime. [ 12.077279] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.090697] systemd[1]: Detected virtualization kvm. [ 12.094923] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.622624] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.626776] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.641085] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.649169] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.653785] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.666237] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.672165] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ 12.731849] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.163873] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.467775] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.498520] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.606888] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.618200] EDAC sbridge: Ver: 1.1.2 [ 15.378900] Key type dns_resolver registered [ 15.715261] NFS: Registering the id_resolver key type [ 15.717863] Key type id_resolver registered [ 15.719607] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg446-server login: [ 38.943398] libcfs: loading out-of-tree module taints kernel. [ 38.960135] Key type ._llcrypt registered [ 38.961221] Key type .llcrypt registered [ 38.996283] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 46.348706] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 46.863376] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 46.870015] alg: No test for adler32 (adler32-zlib) [ 47.839504] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 48.124160] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 49.727153] Key type lgssc registered [ 50.248169] Lustre: Echo OBD driver; http://www.lustre.org/ [ 55.994731] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 68.950917] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 73.076427] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 73.085056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 74.163200] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 74.173100] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 74.207837] Lustre: lustre-MDT0000: new disk, initializing [ 74.228822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 74.240571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 75.515904] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 80.269648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 80.301290] Lustre: 6484:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 80.311714] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 80.313411] Lustre: Skipped 1 previous similar message [ 80.350552] Lustre: lustre-MDT0001: new disk, initializing [ 80.370213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 80.378921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 80.381559] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 81.605450] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 83.739428] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 86.636455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 86.724317] Lustre: lustre-OST0000: new disk, initializing [ 86.726093] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 86.728727] Lustre: 8422:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 86.746366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 88.465822] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 88.938386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 88.940779] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 88.962482] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 93.613735] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 93.653277] Lustre: lustre-OST0001: new disk, initializing [ 93.654902] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 93.657041] Lustre: 9493:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 93.673909] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 95.447440] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 97.769938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 97.773482] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 97.785874] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 100.980529] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 107.426167] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 109.348202] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing check_logdir /tmp/testlogs/ [ 110.763635] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing yml_node [ 112.173836] Lustre: DEBUG MARKER: Client: 2.17.54.84 [ 112.947794] Lustre: DEBUG MARKER: MDS: 2.17.54.84 [ 113.768284] Lustre: DEBUG MARKER: OSS: 2.17.54.84 [ 114.308532] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Jul 4 03:12:40 EDT 2026 [ 119.352451] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 119.811831] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 131.551501] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 131.554308] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 131.559062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 133.599485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 133.600023] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 133.601386] Lustre: Skipped 1 previous similar message [ 133.607319] Lustre: Skipped 1 previous similar message [ 137.620975] Lustre: server umount lustre-MDT0000 complete [ 138.806899] LustreError: 6519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783149184 with bad export cookie 57746910852452229 [ 138.808731] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 138.810103] LustreError: 6519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 143.839670] LustreError: 11212:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 143.839782] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 143.840356] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 143.840363] Lustre: Skipped 2 previous similar messages [ 143.848652] LustreError: 11212:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 143.852306] Lustre: Skipped 2 previous similar messages [ 144.992855] Lustre: server umount lustre-MDT0001 complete [ 152.384343] Lustre: server umount lustre-OST0000 complete [ 159.756868] Lustre: server umount lustre-OST0001 complete [ 161.810832] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 165.101833] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 176.648867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 176.726169] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 176.738697] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 176.775250] Lustre: lustre-MDT0000: new disk, initializing [ 176.795422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 176.802079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 178.033167] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 181.956208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 181.999826] Lustre: 15036:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 182.109114] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 183.351897] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 186.800807] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 188.534752] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 190.667152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 190.757960] Lustre: lustre-OST0000: new disk, initializing [ 190.759642] Lustre: Skipped 1 previous similar message [ 190.761100] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 190.762571] Lustre: Skipped 2 previous similar messages [ 190.764290] Lustre: 17003:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 190.791166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 190.793947] Lustre: Skipped 1 previous similar message [ 191.864366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 191.869671] Lustre: Skipped 1 previous similar message [ 191.871636] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 191.897308] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 192.544684] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 195.992774] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 197.375372] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 197.441934] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 198.820062] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 198.892657] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 201.617069] Lustre: server umount lustre-OST0000 complete [ 207.329193] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 207.330338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 207.332032] Lustre: Skipped 1 previous similar message [ 207.336243] Lustre: Skipped 2 previous similar messages [ 212.448170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 212.450864] Lustre: Skipped 1 previous similar message [ 217.055119] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 217.111897] Lustre: server umount lustre-MDT0000 complete [ 217.568479] LustreError: 15046:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 217.572988] LustreError: 15046:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 218.388215] LustreError: 17002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783149264 with bad export cookie 57746910852453748 [ 218.391333] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 218.392691] LustreError: 17002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 228.885599] Lustre: server umount lustre-MDT0001 complete [ 230.699128] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 03:14:36 (1783149276) [ 236.717158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 236.874968] LustreError: 19874:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 236.899461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 238.352585] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 241.257417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 242.859626] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 245.293199] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 247.076459] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 247.827799] Lustre: Failing over lustre-MDT0000 [ 247.934265] Lustre: server umount lustre-MDT0000 complete [ 251.872883] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 251.873423] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 251.873863] LustreError: 19874:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 251.877515] Lustre: Skipped 1 previous similar message [ 258.017113] LustreError: 19893:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 258.021530] LustreError: 19893:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 260.917962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 260.960719] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 262.335814] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 266.110440] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 266.208499] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 266.211385] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 266.217858] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 268.097083] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 268.097800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 268.101461] Lustre: Skipped 1 previous similar message [ 274.259601] Lustre: server umount lustre-MDT0000 complete [ 275.463248] LustreError: 19854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783149321 with bad export cookie 57746910852455162 [ 275.465924] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 275.466940] LustreError: 19854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 275.576595] Lustre: server umount lustre-MDT0001 complete [ 278.449357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 278.629968] LustreError: 23204:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 278.654592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 278.657935] Lustre: Skipped 2 previous similar messages [ 280.067117] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 282.972764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 284.484242] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 287.106744] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 288.979262] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 289.704932] Lustre: Failing over lustre-MDT0000 [ 289.813314] Lustre: server umount lustre-MDT0000 complete [ 293.344547] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 293.344799] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 293.350993] Lustre: Skipped 1 previous similar message [ 299.488270] LustreError: 23204:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 299.494102] LustreError: 23204:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 302.830790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 302.874827] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 304.362504] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 308.193312] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 308.195712] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 308.199594] Lustre: Skipped 1 previous similar message [ 308.200846] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 308.205593] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 313.312092] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 313.315530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 313.317797] Lustre: Skipped 3 previous similar messages [ 316.316886] Lustre: server umount lustre-MDT0000 complete [ 317.592277] LustreError: 23183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783149363 with bad export cookie 57746910852456100 [ 317.595704] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 317.597774] LustreError: 23183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 317.718174] Lustre: server umount lustre-MDT0001 complete [ 320.631892] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 03:16:06 (1783149366) [ 325.953764] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 329.438724] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 342.508938] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 346.187651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 346.290453] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 346.300266] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 346.339578] Lustre: lustre-MDT0000: new disk, initializing [ 346.363035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 346.365112] Lustre: Skipped 2 previous similar messages [ 346.370022] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 347.685654] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 351.904209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 351.961702] Lustre: 29566:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 352.082253] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 353.492262] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 357.075781] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 358.842059] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 361.133633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 361.254635] Lustre: 31535:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 362.563864] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 362.590822] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 362.596483] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 363.202751] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 366.822461] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 368.708638] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 372.705119] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 372.705518] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 372.707404] LustreError: Skipped 1 previous similar message [ 372.711016] Lustre: Skipped 2 previous similar messages [ 372.711674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 372.715535] Lustre: Skipped 3 previous similar messages [ 378.283690] Lustre: server umount lustre-OST0000 complete [ 382.945672] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 385.630633] Lustre: server umount lustre-MDT0000 complete [ 386.868502] LustreError: 29559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783149432 with bad export cookie 57746910852456506 [ 386.872119] LustreError: 29559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 386.872912] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 387.049508] Lustre: server umount lustre-MDT0001 complete [ 390.733790] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 391.765270] Key type lgssc unregistered [ 391.889349] LNet: 33485:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 391.891965] LNetError: 33485:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 391.902262] LNet: Removed LNI 192.168.204.146@tcp [ 392.275111] Key type .llcrypt unregistered [ 392.276316] Key type ._llcrypt unregistered [ 400.895514] Key type ._llcrypt registered [ 400.896774] Key type .llcrypt registered [ 400.932000] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 401.323065] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 401.355722] alg: No test for adler32 (adler32-zlib) [ 402.226644] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 402.317678] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 403.895126] Key type lgssc registered [ 404.238529] Lustre: Echo OBD driver; http://www.lustre.org/ [ 406.725495] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 406.731483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 407.879741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 409.158034] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 411.960052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 412.083483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 413.368751] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 415.640211] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 417.403808] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 419.603315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 419.736350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 421.592207] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 422.821895] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 423.900954] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 438.714392] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 03:18:04 (1783149484) [ 439.328283] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 439.933093] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 442.012735] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 03:18:07 (1783149487) [ 444.811499] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 03:18:10 (1783149490) [ 445.142055] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 447.801497] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 447.805033] Lustre: Skipped 2 previous similar messages [ 448.321872] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 03:18:14 (1783149494) [ 455.066435] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 455.068902] Lustre: Skipped 10 previous similar messages [ 455.941356] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 03:18:21 (1783149501) [ 456.887790] Lustre: *** cfs_fail_loc=131b, val=2*** [ 458.091810] Lustre: *** cfs_fail_loc=131b, val=2*** [ 460.725676] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 03:18:26 (1783149506) [ 464.922676] Lustre: Setting parameter general.jobid_name=jobid_name=jobid_name=%e.root in log params [ 464.926285] Lustre: Skipped 7 previous similar messages [ 465.520073] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 03:18:31 (1783149511) [ 473.057265] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 473.058859] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 473.059457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 473.062796] Lustre: Skipped 1 previous similar message [ 473.066505] Lustre: Skipped 1 previous similar message [ 475.534255] Lustre: server umount lustre-OST0000 complete [ 478.176542] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 478.180854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 478.184284] Lustre: Skipped 1 previous similar message [ 482.903213] Lustre: server umount lustre-MDT0000 complete [ 483.296224] LustreError: 34870:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 483.301059] LustreError: 34870:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 484.143194] LustreError: 34856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783149530 with bad export cookie 5638404476575297520 [ 484.146436] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 484.146758] LustreError: 34856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 484.266943] Lustre: server umount lustre-MDT0001 complete [ 487.804129] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 488.892280] Key type lgssc unregistered [ 489.029488] LNet: 42708:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 489.032595] LNetError: 42708:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 489.045441] LNet: Removed LNI 192.168.204.146@tcp [ 489.372599] Key type .llcrypt unregistered [ 489.373673] Key type ._llcrypt unregistered [ 495.447953] Key type ._llcrypt registered [ 495.448989] Key type .llcrypt registered [ 495.493341] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 501.701801] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 502.156677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 502.163981] alg: No test for adler32 (adler32-zlib) [ 503.019576] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 503.111687] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 504.695147] Key type lgssc registered [ 505.089327] Lustre: Echo OBD driver; http://www.lustre.org/ [ 516.640405] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 516.647385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 517.745541] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 517.757263] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 517.797613] Lustre: lustre-MDT0000: new disk, initializing [ 517.820264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 517.826561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 519.172609] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 523.224672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 523.264889] Lustre: 45831:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 523.286539] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 523.288179] Lustre: Skipped 1 previous similar message [ 523.325095] Lustre: lustre-MDT0001: new disk, initializing [ 523.361681] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 523.374241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 523.378959] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 524.680642] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 528.280956] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 530.020157] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 532.236572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 532.361870] Lustre: lustre-OST0000: new disk, initializing [ 532.363637] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 532.366069] Lustre: 47800:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 532.396948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 534.346351] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 537.077952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 537.082735] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 537.115255] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 537.917819] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 539.294030] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 539.371104] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 540.774537] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 540.860548] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 542.178499] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 542.178731] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 542.180852] LustreError: Skipped 1 previous similar message [ 542.182313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 542.184951] Lustre: Skipped 1 previous similar message [ 543.569278] Lustre: server umount lustre-OST0000 complete [ 547.296842] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 547.297411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 547.300195] Lustre: Skipped 1 previous similar message [ 547.302956] Lustre: Skipped 1 previous similar message [ 552.416301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 552.418548] Lustre: Skipped 2 previous similar messages [ 557.536464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 557.538529] Lustre: Skipped 1 previous similar message [ 559.071109] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 559.138183] Lustre: server umount lustre-MDT0000 complete [ 560.338169] LustreError: 45826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783149606 with bad export cookie 3659220068658703861 [ 560.341978] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 560.449431] Lustre: server umount lustre-MDT0001 complete [ 563.640992] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 03:20:09 (1783149609) [ 565.775243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 565.946473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 567.257296] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 570.071192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 570.200192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 571.611525] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 573.928057] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 575.605452] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 577.748898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 579.748510] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 582.073943] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 594.383716] LustreError: 52803:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 594.705940] Lustre: Modifying parameter general.timeout=1 in log params [ 595.875268] Lustre: Modifying parameter general.timeout=5 in log params [ 595.877983] Lustre: Skipped 3 previous similar messages [ 597.910041] Lustre: Modifying parameter general.timeout=12 in log params [ 597.912038] Lustre: Skipped 6 previous similar messages [ 602.117163] Lustre: Modifying parameter general.timeout=26 in log params [ 602.119300] Lustre: Skipped 13 previous similar messages [ 610.357128] Lustre: Modifying parameter general.timeout=55 in log params [ 610.358952] Lustre: Skipped 28 previous similar messages [ 626.399899] Lustre: Modifying parameter general.timeout=111 in log params [ 626.401648] Lustre: Skipped 55 previous similar messages [ 635.029968] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 03:21:20 (1783149680) [ 635.905950] Lustre: 59698:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 637.072746] Lustre: 59895:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 637.077182] Lustre: 59895:0:(mgs_llog.c:1447:mgs_modify_param()) Skipped 1 previous similar message [ 647.913621] Lustre: 61528:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 647.916922] Lustre: 61528:0:(mgs_llog.c:1447:mgs_modify_param()) Skipped 3 previous similar messages [ 647.919790] LustreError: 61528:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 649.234697] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 03:21:34 (1783149694) [ 657.376893] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 657.378379] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 657.379371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 657.386808] Lustre: Skipped 1 previous similar message [ 659.032529] Lustre: server umount lustre-MDT0000 complete [ 660.244490] LustreError: 50238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783149706 with bad export cookie 3659220068658704729 [ 660.249148] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 660.250211] LustreError: 50238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 660.366183] Lustre: server umount lustre-MDT0001 complete [ 671.830543] Lustre: server umount lustre-OST0000 complete [ 680.193698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 680.297632] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 680.307270] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 680.310210] Lustre: Skipped 41 previous similar messages [ 680.392698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 680.394687] Lustre: Skipped 1 previous similar message [ 681.654235] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 684.366019] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 684.401834] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 684.412225] Lustre: 64055:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 685.823648] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 688.089754] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 689.827879] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 692.019840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 692.112826] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 694.284765] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 696.638610] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 697.156722] Lustre: 66618:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 699.362670] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 699.362850] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 699.363820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 699.363824] Lustre: Skipped 2 previous similar messages [ 699.366132] Lustre: Skipped 2 previous similar messages [ 699.368191] LustreError: Skipped 1 previous similar message [ 713.183125] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 713.247271] Lustre: server umount lustre-OST0000 complete [ 714.720711] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 714.725074] Lustre: Skipped 1 previous similar message [ 719.840372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 719.842858] Lustre: Skipped 8 previous similar messages [ 720.809610] Lustre: server umount lustre-MDT0000 complete [ 722.149943] LustreError: 66620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783149768 with bad export cookie 3659220068658707018 [ 722.152537] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 722.154226] LustreError: 66620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 722.276849] Lustre: server umount lustre-MDT0001 complete [ 726.082609] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 727.207728] Key type lgssc unregistered [ 727.341415] LNet: 67824:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 727.344436] LNetError: 67824:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 727.353282] LNet: Removed LNI 192.168.204.146@tcp [ 727.657782] Key type .llcrypt unregistered [ 727.659118] Key type ._llcrypt unregistered [ 736.487805] Key type ._llcrypt registered [ 736.488960] Key type .llcrypt registered [ 736.527816] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 736.871132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 736.891602] alg: No test for adler32 (adler32-zlib) [ 737.745421] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 737.828833] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 739.407142] Key type lgssc registered [ 739.775580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 742.264710] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 742.269844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 743.423094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 744.637412] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 747.463610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 747.672387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 749.009317] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 751.221109] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 752.902467] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 755.005250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 755.180441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 757.063718] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 759.304185] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 772.422841] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 778.208168] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 778.210861] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 778.215267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 779.748059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 779.749942] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 779.752177] Lustre: Skipped 1 previous similar message [ 779.762012] Lustre: Skipped 1 previous similar message [ 781.021974] Lustre: server umount lustre-MDT0000 complete [ 782.201202] LustreError: 69205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783149828 with bad export cookie 9683348169066791818 [ 782.204943] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 782.205832] LustreError: 69205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 782.308146] Lustre: server umount lustre-MDT0001 complete [ 793.817217] Lustre: server umount lustre-OST0000 complete [ 802.648437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 802.750680] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 802.759876] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 802.851727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 804.199803] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 807.113691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 807.150451] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 807.160231] Lustre: 74112:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 807.329209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 808.656911] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 811.012228] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 812.923774] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 815.108554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 815.186182] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 817.322341] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 819.780762] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 820.298501] Lustre: 76684:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 820.945438] Lustre: Modifying parameter general.jobid_var=disable in log params [ 826.341114] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 826.341840] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 826.342992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 826.343031] Lustre: Skipped 1 previous similar message [ 826.343246] LustreError: Skipped 1 previous similar message [ 826.357590] Lustre: Skipped 1 previous similar message [ 828.304030] Lustre: server umount lustre-OST0000 complete [ 831.458356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 831.459512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 831.461877] Lustre: Skipped 1 previous similar message [ 831.467443] Lustre: Skipped 2 previous similar messages [ 835.800234] Lustre: server umount lustre-MDT0000 complete [ 836.575965] LustreError: 74124:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 836.581786] LustreError: 74124:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 837.049067] LustreError: 75976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783149883 with bad export cookie 9683348169066792637 [ 837.052958] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 837.053376] LustreError: 75976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 837.177108] Lustre: server umount lustre-MDT0001 complete [ 840.900590] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 841.884394] Key type lgssc unregistered [ 842.008368] LNet: 77889:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 842.010963] LNetError: 77889:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 842.022340] LNet: Removed LNI 192.168.204.146@tcp [ 842.330587] Key type .llcrypt unregistered [ 842.332310] Key type ._llcrypt unregistered [ 851.551452] Key type ._llcrypt registered [ 851.552766] Key type .llcrypt registered [ 851.590247] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 851.902597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 851.945050] alg: No test for adler32 (adler32-zlib) [ 852.796382] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 852.882730] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 854.463235] Key type lgssc registered [ 854.890751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 857.502836] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 857.508190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 858.666167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 859.976967] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 862.806069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 863.030772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 864.389846] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 866.787410] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 868.669021] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 870.933642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 871.147584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 873.410265] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 875.991102] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 888.524499] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 889.132829] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 889.136341] Lustre: Skipped 1 previous similar message [ 890.299146] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 890.302444] Lustre: Skipped 3 previous similar messages [ 892.364592] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 892.366642] Lustre: Skipped 6 previous similar messages [ 896.407438] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 896.409457] Lustre: Skipped 13 previous similar messages [ 904.567949] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 904.570131] Lustre: Skipped 27 previous similar messages [ 919.719351] Lustre: 87363:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 920.539159] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 03:26:06 (1783149966) [ 921.022973] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 921.542289] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 03:26:07 (1783149967) [ 923.615862] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 923.618062] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 923.621922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 924.643278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 924.643371] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 924.643687] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 928.521179] Lustre: server umount lustre-OST0000 complete [ 929.761372] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 929.761420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 929.762443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 929.762448] Lustre: Skipped 1 previous similar message [ 929.765184] Lustre: Skipped 1 previous similar message [ 934.880740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 934.882867] Lustre: Skipped 2 previous similar messages [ 935.834157] Lustre: server umount lustre-MDT0000 complete [ 937.115290] LustreError: 79258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783149983 with bad export cookie 12014247852332457668 [ 937.120511] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 937.121128] LustreError: 79258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 937.237663] Lustre: server umount lustre-MDT0001 complete [ 940.836063] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 941.883168] Key type lgssc unregistered [ 942.012344] LNet: 88609:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 942.015305] LNetError: 88609:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 942.024484] LNet: Removed LNI 192.168.204.146@tcp [ 942.347765] Key type .llcrypt unregistered [ 942.348966] Key type ._llcrypt unregistered [ 943.814897] Key type ._llcrypt registered [ 943.815949] Key type .llcrypt registered [ 943.854546] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 945.215981] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules [ 945.545594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 945.598100] alg: No test for adler32 (adler32-zlib) [ 946.459441] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 946.549080] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 948.127151] Key type lgssc registered [ 948.559751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 949.003634] LustreError: 89762:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 956.700637] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 960.027993] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 970.267367] LustreError: 89762:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 970.274399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 971.445297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 972.697829] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 975.604202] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 03:27:01 (1783150021) [ 977.115296] LustreError: 90820:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 977.119103] LustreError: 90820:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 977.122399] LustreError: 90820:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 977.242884] Lustre: server umount lustre-MDT0000 complete [ 981.710983] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 982.702647] Key type lgssc unregistered [ 982.831413] LNet: 92313:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 982.834410] LNetError: 92313:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 982.844232] LNet: Removed LNI 192.168.204.146@tcp [ 983.123833] Key type .llcrypt unregistered [ 983.124964] Key type ._llcrypt unregistered [ 990.595465] Key type ._llcrypt registered [ 990.597543] Key type .llcrypt registered [ 990.647279] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 991.165556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 991.175889] alg: No test for adler32 (adler32-zlib) [ 992.030691] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 992.119782] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 993.695177] Key type lgssc registered [ 994.061263] Lustre: Echo OBD driver; http://www.lustre.org/ [ 996.594271] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 996.601073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 997.770676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 999.158492] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1002.048229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1002.264462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1003.561830] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1005.888257] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1007.659681] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1009.943717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1010.158609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1012.353666] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1014.914817] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1046.126211] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 03:28:11 (1783150091) [ 1046.786976] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1047.499469] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 03:28:13 (1783150093) [ 1053.154942] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1053.156779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1053.159317] Lustre: Skipped 2 previous similar messages [ 1053.162655] Lustre: Skipped 2 previous similar messages [ 1054.748119] Lustre: server umount lustre-MDT0000 complete [ 1056.120538] LustreError: 93488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783150102 with bad export cookie 534563260188560867 [ 1056.122805] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1056.124840] LustreError: 93488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1056.251542] Lustre: server umount lustre-MDT0001 complete [ 1067.072422] Lustre: server umount lustre-OST0000 complete [ 1071.119705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1071.304791] LustreError: 97796:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1071.313276] LustreError: 97796:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1071.384396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1072.839150] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1075.846576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1076.089990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1077.480267] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1080.011903] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1081.944511] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1086.200057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1086.295531] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1086.299463] LustreError: 97785:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1086.313972] LustreError: 99885:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1086.321045] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1086.324172] LustreError: 99878:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 1086.326307] LustreError: 99878:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 1086.328216] LustreError: 99878:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1086.355983] Lustre: server umount lustre-OST0000 complete [ 1086.357366] LustreError: 99878:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1089.421923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1089.502978] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1089.507892] LustreError: 97784:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1089.518396] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1089.521584] LustreError: 100344:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 1089.524127] LustreError: 100344:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 1089.526141] LustreError: 100344:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1089.547323] Lustre: server umount lustre-OST0000 complete [ 1089.548723] LustreError: 100344:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1092.996427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1093.078631] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1093.081279] Lustre: Found index 0 for lustre-OST0000, updating log [ 1093.084179] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1093.117040] Lustre: lustre-OST0000: new disk, initializing [ 1093.119164] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1093.122971] Lustre: 100873:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1093.218982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1095.219597] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1096.648225] Lustre: Failing over lustre-OST0000 [ 1096.722374] Lustre: server umount lustre-OST0000 complete [ 1101.793834] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1101.794408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1101.795580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1101.809558] Lustre: Skipped 1 previous similar message [ 1106.912824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1106.916448] Lustre: Skipped 2 previous similar messages [ 1107.973104] Lustre: server umount lustre-MDT0000 complete [ 1109.331316] LustreError: 97776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783150155 with bad export cookie 534563260188563597 [ 1109.335457] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1109.338095] LustreError: 97776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1109.445139] Lustre: server umount lustre-MDT0001 complete [ 1112.145444] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 03:29:17 (1783150157) [ 1117.262870] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1120.875518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1121.010148] LustreError: 103212:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1121.033198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1122.375482] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1125.063871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1126.606013] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1127.541165] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1129.676614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1129.961155] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 1130.977927] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1130.989526] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1130.989694] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 1130.992516] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 1131.884163] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1134.709754] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1134.745629] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1134.782357] Lustre: lustre-OST0001: new disk, initializing [ 1134.785146] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1134.788972] Lustre: 105530:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1136.724549] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1137.984730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1137.988562] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1138.040070] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1139.695650] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1143.218964] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1145.311970] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 1145.314301] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1145.318857] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1148.387787] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1148.394077] Lustre: Skipped 1 previous similar message [ 1150.611428] Lustre: server umount lustre-MDT0001 complete [ 1153.145782] LustreError: 104888:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1153.648457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1153.683351] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1153.685032] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1153.699506] Lustre: 103201:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1153.937895] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1153.940906] Lustre: Skipped 3 previous similar messages [ 1155.346853] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1155.622655] LustreError: 107318:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1155.626647] LustreError: 107318:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1155.631584] LustreError: 103205:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1155.662403] LustreError: 107340:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.146@tcp: cfg command failed: rc = -17 [ 1155.667096] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1155.667096] [ 1158.372600] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 03:30:04 (1783150204) [ 1159.143651] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1164.086980] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1169.998784] Lustre: 109130:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1177.709587] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1178.926731] Lustre: 110262:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1225.696557] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1225.697151] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1225.697187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1225.697213] Lustre: Skipped 4 previous similar messages [ 1225.701432] Lustre: Skipped 2 previous similar messages [ 1230.120893] Lustre: server umount lustre-MDT0000 complete [ 1230.815738] LustreError: 103208:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1230.821501] LustreError: 103208:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 1231.354244] LustreError: 103194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783150277 with bad export cookie 534563260188564514 [ 1231.356193] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1231.359298] LustreError: 103194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1231.474553] Lustre: server umount lustre-MDT0001 complete [ 1242.120338] Lustre: server umount lustre-OST0000 complete [ 1253.521541] Lustre: server umount lustre-OST0001 complete [ 1255.318525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1258.336416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1261.054853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1264.195629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1270.404413] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1273.919721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1273.925409] Lustre: lustre-MDT0000: reset Object Index mappings [ 1274.069870] LustreError: 114186:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1274.092699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1275.430515] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1278.306369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1278.315575] Lustre: lustre-MDT0001: reset Object Index mappings [ 1279.952420] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1280.899466] Lustre: 115334:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1283.047591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1284.333782] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 1285.239952] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1287.891865] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1289.131684] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 1289.132773] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 1289.136743] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 1289.801439] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1292.935725] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1294.138889] Lustre: 117210:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1308.866076] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 03:32:34 (1783150354) [ 1339.872137] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1339.876360] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1339.882700] Lustre: Skipped 1 previous similar message [ 1339.885459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1339.888572] Lustre: Skipped 3 previous similar messages [ 1344.415866] Lustre: server umount lustre-MDT0000 complete [ 1345.504368] LustreError: 114923:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1345.510921] LustreError: 114923:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1345.586022] LustreError: 114167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783150391 with bad export cookie 534563260188611421 [ 1345.590784] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1351.927753] Lustre: server umount lustre-MDT0001 complete [ 1359.303759] Lustre: server umount lustre-OST0000 complete [ 1366.743721] Lustre: server umount lustre-OST0001 complete [ 1368.844107] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 1372.137340] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1383.801391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1383.903954] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1383.914448] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1383.953930] Lustre: lustre-MDT0000: new disk, initializing [ 1383.981895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1385.348946] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1388.371024] Lustre: server umount lustre-MDT0000 complete [ 1391.554339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1391.676413] Lustre: Found index 0 for lustre-MDT0000, updating log [ 1391.681135] Lustre: 121463:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1391.686203] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 1393.176114] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1394.897639] Lustre: server umount lustre-MDT0000 complete [ 1397.811302] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 03:34:03 (1783150443) [ 1398.365591] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 1398.994103] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 03:34:04 (1783150444) [ 1404.630857] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1408.252699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1408.432062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1408.435172] Lustre: Skipped 5 previous similar messages [ 1409.922003] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1412.977229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1413.020769] Lustre: 123513:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1413.046207] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1413.048351] Lustre: Skipped 1 previous similar message [ 1413.090739] Lustre: lustre-MDT0001: new disk, initializing [ 1413.139675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1413.143035] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1414.526342] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1417.003617] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1419.731451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1419.870211] Lustre: lustre-OST0000: new disk, initializing [ 1419.872515] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1419.875992] Lustre: 125090:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1421.625714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1421.630120] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1421.667166] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1422.188853] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1426.358969] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1426.426962] Lustre: 125960:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1428.366267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1428.369918] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1428.423178] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1428.520757] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1432.997057] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1434.223143] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1579.643459] Lustre: 132338:0:(osd_internal.h:1444:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 1579.647014] Lustre: 132338:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 1579.649224] Lustre: 132338:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 1579.652154] Lustre: 132338:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 1579.655904] Lustre: 132338:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 1579.658374] Lustre: 132338:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1645.737800] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 03:38:11 (1783150691) [ 1658.849872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1658.850063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1658.853369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1658.857094] Lustre: Skipped 7 previous similar messages [ 1658.859079] Lustre: Skipped 7 previous similar messages [ 1663.541298] Lustre: server umount lustre-MDT0000 complete [ 1663.969095] LustreError: 126758:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1663.977756] LustreError: 126758:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 1664.833194] LustreError: 124720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783150710 with bad export cookie 534563260188664593 [ 1664.840427] LustreError: 124720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1664.841911] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1679.327243] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1679.476554] Lustre: server umount lustre-MDT0001 complete [ 1687.182890] Lustre: server umount lustre-OST0000 complete [ 1702.879190] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1702.985477] Lustre: server umount lustre-OST0001 complete [ 1705.214512] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 1708.314340] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1722.851312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1723.005133] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1723.020903] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1723.025166] Lustre: Skipped 1 previous similar message [ 1723.077596] Lustre: lustre-MDT0000: new disk, initializing [ 1723.079366] Lustre: Skipped 1 previous similar message [ 1723.116520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1723.121013] Lustre: Skipped 3 previous similar messages [ 1723.129706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1725.073729] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1729.428408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1729.480095] Lustre: 135349:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1729.607011] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1731.287651] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1735.446797] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1737.495951] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1740.312523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1740.443492] Lustre: lustre-OST0000: new disk, initializing [ 1740.446445] Lustre: Skipped 1 previous similar message [ 1740.449660] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1740.453479] Lustre: Skipped 2 previous similar messages [ 1740.457393] Lustre: 137319:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1741.851353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1741.856625] Lustre: Skipped 1 previous similar message [ 1741.860630] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1741.920725] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1743.275295] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1747.581799] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1751.283918] Lustre: lustre-MDD0000: changelog on [ 1752.247992] Lustre: lustre-MDD0001: changelog on [ 2124.333115] Lustre: lustre-MDD0001: changelog off [ 2124.966693] Lustre: 137991:0:(llog_cat.c:977:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 2125.266622] Lustre: lustre-MDD0000: changelog off [ 2130.914076] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2130.917120] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2130.918861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2130.921212] Lustre: Skipped 4 previous similar messages [ 2130.923114] Lustre: Skipped 8 previous similar messages [ 2141.151189] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2141.246407] Lustre: server umount lustre-OST0000 complete [ 2148.984351] Lustre: server umount lustre-MDT0000 complete [ 2150.244367] LustreError: 135342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783151196 with bad export cookie 534563260189429364 [ 2150.247538] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2150.248140] LustreError: 135342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2150.358958] Lustre: server umount lustre-MDT0001 complete [ 2154.258258] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 2155.133384] Key type lgssc unregistered [ 2155.258352] LNet: 145711:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2155.260975] LNetError: 145711:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2155.269257] LNet: Removed LNI 192.168.204.146@tcp [ 2155.563084] Key type .llcrypt unregistered [ 2155.564570] Key type ._llcrypt unregistered [ 2161.780767] Key type ._llcrypt registered [ 2161.781921] Key type .llcrypt registered [ 2161.813887] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 2167.385739] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2167.671277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2167.679989] alg: No test for adler32 (adler32-zlib) [ 2168.519154] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 2168.601760] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 2170.175083] Key type lgssc registered [ 2170.484674] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2180.455735] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2180.463108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2181.546751] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2181.556456] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2181.594108] Lustre: lustre-MDT0000: new disk, initializing [ 2181.613826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2181.620642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2182.833098] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2186.667769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2186.705508] Lustre: 148885:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2186.728564] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2186.730467] Lustre: Skipped 1 previous similar message [ 2186.765194] Lustre: lustre-MDT0001: new disk, initializing [ 2186.795234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2186.805600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2186.808745] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2188.057837] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2192.539627] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2194.545991] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2197.249172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2197.357868] Lustre: lustre-OST0000: new disk, initializing [ 2197.360890] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2197.365466] Lustre: 150853:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2197.401865] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2199.042251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2199.046383] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2199.094580] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2199.482850] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2203.456191] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2205.950320] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_module obdecho/obdecho [ 2206.701907] LustreError: 151857:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 2206.707401] LustreError: 151857:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 2207.458156] LustreError: 151953:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 2212.320541] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2212.325651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2212.334652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2214.371468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2214.372770] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2214.375486] Lustre: Skipped 1 previous similar message [ 2214.383445] Lustre: Skipped 1 previous similar message [ 2218.032238] Lustre: server umount lustre-MDT0000 complete [ 2219.833635] LustreError: 150852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783151265 with bad export cookie 13877597089499354129 [ 2219.836680] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2219.840058] LustreError: 150852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2219.993711] Lustre: server umount lustre-MDT0001 complete [ 2231.989940] Lustre: server umount lustre-OST0000 complete [ 2241.245872] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 2245.233967] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2259.155900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2259.250264] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2259.262026] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2259.297607] Lustre: lustre-MDT0000: new disk, initializing [ 2259.319743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2259.326216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2260.774669] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2265.459633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2265.507737] Lustre: 155659:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2265.535062] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2265.536859] Lustre: Skipped 1 previous similar message [ 2265.570270] Lustre: lustre-MDT0001: new disk, initializing [ 2265.605802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2265.617890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2265.621419] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2267.006151] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2270.946095] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2273.213644] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2276.257448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2276.360965] Lustre: lustre-OST0000: new disk, initializing [ 2276.362878] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2276.365305] Lustre: 157628:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2276.393585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2278.755629] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2282.989560] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2284.970740] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2286.071483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2286.074744] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2286.105834] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2287.089588] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2288.814385] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2288.919793] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2291.174269] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2291.174631] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2291.179352] LustreError: Skipped 1 previous similar message [ 2291.191364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2291.196624] Lustre: Skipped 2 previous similar messages [ 2295.953344] Lustre: server umount lustre-OST0000 complete [ 2301.408908] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2301.408984] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2301.409505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2301.416852] Lustre: Skipped 2 previous similar messages [ 2303.461987] Lustre: server umount lustre-MDT0000 complete [ 2304.709267] LustreError: 159184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783151350 with bad export cookie 13877597089499355074 [ 2304.712630] LustreError: 159184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2304.712835] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2304.819784] Lustre: server umount lustre-MDT0001 complete [ 2306.079763] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 03:49:11 (1783151351) [ 2306.684811] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 2307.366871] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 03:49:13 (1783151353) [ 2309.888784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2310.038623] LustreError: 160053:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2310.044930] LustreError: 160053:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2310.060375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2311.348382] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2314.868291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2316.440232] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2318.848627] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2320.656897] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2322.990669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2324.914080] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2327.438888] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2340.833284] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2340.834160] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2340.837242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2340.840272] Lustre: Skipped 2 previous similar messages [ 2340.844162] Lustre: Skipped 3 previous similar messages [ 2345.821194] Lustre: server umount lustre-MDT0000 complete [ 2345.952566] LustreError: 160054:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2345.958670] LustreError: 160054:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2346.963175] LustreError: 160034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783151393 with bad export cookie 13877597089499355942 [ 2346.966957] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2346.967213] LustreError: 160034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2348.152262] LustreError: 162540:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2351.071805] LustreError: 162540:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2351.072098] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2351.082985] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2351.086034] Lustre: Skipped 2 previous similar messages [ 2356.191726] LustreError: 160789:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2356.196567] LustreError: 160789:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2361.311090] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2361.398764] Lustre: server umount lustre-MDT0001 complete [ 2363.457225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2367.582284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2367.666168] LustreError: 163913:0:(llog.c:1646:llog_backup()) MGC192.168.204.146@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 2367.669720] Lustre: 163913:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.146@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 2387.936482] LustreError: 163918:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2387.940752] LustreError: 163918:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2404.472261] LustreError: 163939:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2404.477564] LustreError: 163939:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 2428.895285] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 2428.899132] Lustre: 163913:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.204.146@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 2438.112676] LustreError: 163939:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2438.123223] LustreError: 163939:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 21 previous similar messages [ 2469.420631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2469.423744] Lustre: Skipped 2 previous similar messages [ 2470.789148] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2472.419435] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2473.534564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2492.899805] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xc0972106b79ab471 [ 2492.904073] Lustre: MGC192.168.204.146@tcp: Connection restored to 0@lo (at 0@lo) [ 2494.431884] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2496.659815] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2497.205093] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2498.023950] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2499.151265] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2499.747660] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2506.710550] Lustre: server umount lustre-MDT0000 complete [ 2507.955530] LustreError: 165545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783151554 with bad export cookie 13877597089499362417 [ 2507.959335] LustreError: 165545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2507.960518] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2508.058605] Lustre: server umount lustre-MDT0001 complete [ 2529.694449] Lustre: server umount lustre-OST0000 complete [ 2536.090737] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 2539.218107] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2550.795807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2550.876202] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2550.888051] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2550.926261] Lustre: lustre-MDT0000: new disk, initializing [ 2550.946674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2550.948822] Lustre: Skipped 1 previous similar message [ 2550.953476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2552.156284] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2556.078315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2556.117408] Lustre: 168999:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2556.146707] LustreError: 169008:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2556.151829] LustreError: 169008:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 41 previous similar messages [ 2556.219372] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2557.425204] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2561.108526] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2563.117374] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2565.499549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2565.602159] Lustre: 170966:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2567.412815] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2571.179727] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2572.278950] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2572.314895] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2572.523179] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2572.588038] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2573.864923] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2573.928249] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2577.377666] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2577.378572] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2577.379127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2577.379133] Lustre: Skipped 4 previous similar messages [ 2577.381212] Lustre: Skipped 1 previous similar message [ 2580.621529] Lustre: server umount lustre-OST0000 complete [ 2596.319128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2596.388286] Lustre: server umount lustre-MDT0000 complete [ 2597.571047] LustreError: 170965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783151643 with bad export cookie 13877597089499379217 [ 2597.574332] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2597.574514] LustreError: 170965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2597.694184] Lustre: server umount lustre-MDT0001 complete [ 2600.559926] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 03:54:06 (1783151646) [ 2602.730302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2604.468531] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2607.232749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2608.649039] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2610.931320] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2612.651021] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2614.781986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2616.728716] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2619.763968] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2634.382267] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 03:54:40 (1783151680) [ 2637.793558] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2637.795575] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2637.798306] LustreError: Skipped 3 previous similar messages [ 2637.804150] Lustre: Skipped 2 previous similar messages [ 2637.805174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2637.813989] Lustre: Skipped 9 previous similar messages [ 2641.509504] Lustre: server umount lustre-OST0000 complete [ 2657.247196] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2657.332740] Lustre: server umount lustre-MDT0000 complete [ 2659.003703] LustreError: 173358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783151705 with bad export cookie 13877597089499380099 [ 2659.009699] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2659.174408] Lustre: server umount lustre-MDT0001 complete [ 2663.080025] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 2664.469151] Key type lgssc unregistered [ 2664.615662] LNet: 177125:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2664.620419] LNetError: 177125:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2664.630631] LNet: Removed LNI 192.168.204.146@tcp [ 2665.030163] Key type .llcrypt unregistered [ 2665.031978] Key type ._llcrypt unregistered [ 2672.565323] Key type ._llcrypt registered [ 2672.566806] Key type .llcrypt registered [ 2672.611441] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2673.119279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2673.147727] alg: No test for adler32 (adler32-zlib) [ 2674.008164] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 2674.118969] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 2675.703184] Key type lgssc registered [ 2676.012078] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2678.378027] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2678.383670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2693.791486] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2698.911515] LustreError: 178344:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.146@tcp: failed processing log, type 4: rc = -110 [ 2704.031822] LustreError: 178344:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 2704.040508] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 2704.044342] LustreError: MGC192.168.204.146@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 2704.051635] LustreError: 178344:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 2704.057267] LustreError: 178344:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 2704.062147] LustreError: 178344:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2704.095856] Lustre: server umount lustre-OST0000 complete [ 2704.098886] LustreError: 178344:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 2707.484923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2708.626424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2709.921364] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2712.942398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2713.080385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2714.633865] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2717.703976] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2719.844142] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2722.774545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2722.893707] LustreError: 180657:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 2722.899723] Lustre: 180657:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.146@tcp: invalid local config log lustre-OST0000: rc = -22 [ 2722.962856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2725.349212] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2727.931961] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2729.445046] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2729.447500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2729.454535] Lustre: Skipped 2 previous similar messages [ 2729.461292] Lustre: Skipped 2 previous similar messages [ 2734.560421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2734.562928] Lustre: Skipped 2 previous similar messages [ 2739.684832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2739.691307] Lustre: Skipped 2 previous similar messages [ 2743.263111] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2743.327188] Lustre: server umount lustre-MDT0000 complete [ 2744.803613] LustreError: 178845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2744.815248] LustreError: 178845:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2744.901697] LustreError: 178824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783151790 with bad export cookie 4045772233114934592 [ 2744.908796] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2744.910332] LustreError: 178824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2745.084610] Lustre: server umount lustre-MDT0001 complete [ 2756.841153] Lustre: server umount lustre-OST0000 complete [ 2760.657119] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 2764.553187] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2779.034791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2779.137097] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2779.151377] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2779.189349] Lustre: lustre-MDT0000: new disk, initializing [ 2779.209841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2779.215972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2780.568205] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2784.828922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2784.887396] Lustre: 184177:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2784.943811] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2784.946982] Lustre: Skipped 1 previous similar message [ 2784.992763] Lustre: lustre-MDT0001: new disk, initializing [ 2785.030679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2785.044712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2785.049222] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2786.479090] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2790.278070] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2792.020828] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2794.293797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2794.392401] Lustre: lustre-OST0000: new disk, initializing [ 2794.393903] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2794.396169] Lustre: 186146:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2794.429448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2794.604893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2794.609829] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2794.650482] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2796.253796] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2799.770777] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2801.054901] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2801.117011] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2802.353212] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2802.412649] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2805.070351] Lustre: server umount lustre-OST0000 complete [ 2810.336585] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2810.337273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2810.342682] Lustre: Skipped 1 previous similar message [ 2815.456123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2815.457880] Lustre: Skipped 2 previous similar messages [ 2816.503183] Lustre: server umount lustre-MDT0000 complete [ 2817.666176] LustreError: 184975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783151863 with bad export cookie 4045772233114935278 [ 2817.668812] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2817.671287] LustreError: 184975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2827.277530] Lustre: server umount lustre-MDT0001 complete [ 2829.919269] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 03:57:55 (1783151875) [ 2835.079908] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 2835.948090] Key type lgssc unregistered [ 2836.072311] LNet: 189088:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2836.074640] LNetError: 189088:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2836.083269] LNet: Removed LNI 192.168.204.146@tcp [ 2836.353917] Key type .llcrypt unregistered [ 2836.355188] Key type ._llcrypt unregistered [ 2842.809712] Key type ._llcrypt registered [ 2842.810788] Key type .llcrypt registered [ 2842.843141] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2843.155827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2843.193602] alg: No test for adler32 (adler32-zlib) [ 2844.042586] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 2844.125158] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 2845.703107] Key type lgssc registered [ 2846.033860] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2848.501659] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2848.507180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2849.638682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2850.861662] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2853.421988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2853.537906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2854.760764] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2857.067661] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2859.111720] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2862.428969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2862.547341] Lustre: *** cfs_fail_loc=131e, val=0*** [ 2862.548869] LustreError: 192143:0:(llog.c:1687:llog_backup()) MGC192.168.204.146@tcp: failed to backup log lustre-OST0000: rc = -28 [ 2862.552705] Lustre: 192143:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.146@tcp: can't backup local config lustre-OST0000: rc = -28 [ 2862.616804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2864.902944] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2867.338711] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2868.705499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2868.707065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2868.710563] Lustre: Skipped 2 previous similar messages [ 2868.712419] Lustre: Skipped 1 previous similar message [ 2873.824331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2873.827611] Lustre: Skipped 3 previous similar messages [ 2874.133572] Lustre: server umount lustre-MDT0000 complete [ 2875.385151] LustreError: 190264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783151921 with bad export cookie 9619305391486937629 [ 2875.388670] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2875.392225] LustreError: 190264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2875.498167] Lustre: server umount lustre-MDT0001 complete [ 2885.970717] Lustre: server umount lustre-OST0000 complete [ 2889.255462] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 2892.880744] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2906.209522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2906.309808] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2906.320963] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2906.357883] Lustre: lustre-MDT0000: new disk, initializing [ 2906.379240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2906.385365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2907.805581] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2912.308794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2912.353822] Lustre: 195662:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2912.376720] Lustre: *** cfs_fail_loc=131e, val=0*** [ 2912.378257] Lustre: Skipped 1 previous similar message [ 2912.379884] LustreError: 196445:0:(llog.c:1687:llog_backup()) MGC192.168.204.146@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 2912.383298] LustreError: 196445:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 2912.386803] Lustre: 196445:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.146@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 2912.396045] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2912.398655] Lustre: Skipped 1 previous similar message [ 2912.434338] Lustre: lustre-MDT0001: new disk, initializing [ 2912.471537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2912.485129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2912.489083] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2913.819581] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2917.898940] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2919.660459] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2922.070510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2922.181084] Lustre: *** cfs_fail_loc=131e, val=0*** [ 2922.182429] Lustre: Skipped 1 previous similar message [ 2922.183888] LustreError: 197604:0:(llog.c:1687:llog_backup()) MGC192.168.204.146@tcp: failed to backup log lustre-OST0000: rc = -28 [ 2922.187448] LustreError: 197604:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 2922.189681] Lustre: 197604:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.146@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 2922.194119] Lustre: 197604:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 2922.206858] Lustre: lustre-OST0000: new disk, initializing [ 2922.210396] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2922.213648] Lustre: 197628:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2922.253332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2922.417929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2922.423891] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2922.468077] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2924.233888] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2927.962500] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2929.456488] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2929.531494] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2930.933782] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2931.011357] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2932.708855] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2932.709153] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2932.714315] LustreError: Skipped 1 previous similar message [ 2932.726608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2932.731696] Lustre: Skipped 1 previous similar message [ 2937.830033] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2937.835314] Lustre: Skipped 1 previous similar message [ 2942.948664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2942.952725] Lustre: Skipped 1 previous similar message [ 2946.015067] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2946.062291] Lustre: server umount lustre-OST0000 complete [ 2948.064871] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2948.068176] Lustre: Skipped 2 previous similar messages [ 2953.184531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2953.187557] Lustre: Skipped 3 previous similar messages [ 2953.309491] Lustre: server umount lustre-MDT0000 complete [ 2954.686791] LustreError: 197627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783152000 with bad export cookie 9619305391486938329 [ 2954.694184] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2954.696351] LustreError: 197627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2954.840149] Lustre: server umount lustre-MDT0001 complete [ 2957.872373] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 04:00:03 (1783152003) [ 2964.577225] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2968.582402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2968.700321] LustreError: 200752:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2968.705363] LustreError: 200752:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2968.711500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2970.482514] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2974.140354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2974.177029] LustreError: 200753:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2976.056474] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2977.425767] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2980.165886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2982.790897] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2986.877432] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2986.982524] Lustre: lustre-OST0001: new disk, initializing [ 2986.985796] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2986.990859] Lustre: 203059:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2987.515717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2987.522161] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2987.578090] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2989.502359] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2995.938860] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3001.262030] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3002.261212] Lustre: DEBUG MARKER: ADD OST3 [ 3004.813643] loop: module loaded [ 3004.844480] loop0: detected capacity change from 0 to 204800000 [ 3004.934357] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3004.937915] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3004.944522] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3007.712580] loop0: detected capacity change from 0 to 204800000 [ 3009.948783] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3010.047282] Lustre: lustre-OST0003: new disk, initializing [ 3010.049133] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 3010.052340] Lustre: 204810:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3010.103482] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 3010.105985] Lustre: Skipped 3 previous similar messages [ 3010.129311] Lustre: DEBUG MARKER: STOP OST3 [ 3011.087655] Lustre: Failing over lustre-OST0003 [ 3011.135395] Lustre: server umount lustre-OST0003 complete [ 3011.919958] LustreError: 202237:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3012.767948] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3016.763508] Lustre: DEBUG MARKER: START OST3 again [ 3017.190817] LustreError: 203839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3017.195909] LustreError: 203839:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3020.108652] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3023.069152] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3025.411338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 3025.418104] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 3025.435350] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 3037.639553] Lustre: Failing over lustre-OST0003 [ 3037.677667] Lustre: server umount lustre-OST0003 complete [ 3037.823323] LustreError: 203839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3037.827828] LustreError: 203839:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3040.740719] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3040.743117] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3040.743394] LustreError: Skipped 1 previous similar message [ 3045.865448] LustreError: 202236:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3045.870683] LustreError: 202236:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3050.981166] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3050.983482] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3050.985266] Lustre: Skipped 2 previous similar messages [ 3050.987562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3056.487366] Lustre: server umount lustre-MDT0000 complete [ 3057.782052] LustreError: 200733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783152103 with bad export cookie 9619305391486939197 [ 3057.785682] LustreError: 200733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3057.786165] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3061.216880] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3061.220835] Lustre: Skipped 1 previous similar message [ 3062.239952] LustreError: 200749:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3062.244255] LustreError: 200749:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3064.106799] Lustre: server umount lustre-MDT0001 complete [ 3065.485836] Lustre: server umount lustre-OST0000 complete [ 3066.745201] Lustre: server umount lustre-OST0001 complete [ 3068.888276] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 3072.340755] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3089.975416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3090.098769] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3090.114223] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3090.160583] Lustre: lustre-MDT0000: new disk, initializing [ 3090.186367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3090.188990] Lustre: Skipped 1 previous similar message [ 3090.195027] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3091.994410] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3097.595198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3097.667246] Lustre: 210212:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3097.824880] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3099.668479] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3104.658481] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3107.413857] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3110.694729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3110.861451] Lustre: lustre-OST0000: new disk, initializing [ 3110.863962] Lustre: Skipped 1 previous similar message [ 3110.866603] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3110.869772] Lustre: Skipped 2 previous similar messages [ 3110.873050] Lustre: 212182:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3112.699983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3112.702700] Lustre: Skipped 1 previous similar message [ 3112.704635] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3112.742966] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3113.217124] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3117.872558] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3119.172073] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3119.230840] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3120.380012] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3120.440388] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3123.170103] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3123.170791] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3123.171839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3123.171845] Lustre: Skipped 11 previous similar messages [ 3123.172304] LustreError: Skipped 1 previous similar message [ 3123.179602] Lustre: Skipped 1 previous similar message [ 3127.143548] Lustre: server umount lustre-OST0000 complete [ 3143.135158] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3143.227101] Lustre: server umount lustre-MDT0000 complete [ 3143.649215] LustreError: 210224:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3143.658755] LustreError: 210224:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3144.997529] LustreError: 210205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783152191 with bad export cookie 9619305391486941458 [ 3145.004850] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3145.005913] LustreError: 210205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3145.170357] Lustre: server umount lustre-MDT0001 complete [ 3146.870295] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 04:03:12 (1783152192) [ 3153.473981] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 3157.898425] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3174.539176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3174.666729] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3174.681770] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3174.741460] Lustre: lustre-MDT0000: new disk, initializing [ 3174.782255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3176.560581] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3182.224681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3182.284611] Lustre: 216785:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3182.426481] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3184.159809] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3187.923142] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3190.493510] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3193.653649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3193.805370] Lustre: 218754:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3195.148805] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3196.436980] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3201.334610] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3203.021279] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3203.089160] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3204.708709] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3204.821304] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3210.721901] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3210.722755] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3210.725657] LustreError: Skipped 1 previous similar message [ 3210.732465] Lustre: Skipped 2 previous similar messages [ 3210.733504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3210.738655] Lustre: Skipped 6 previous similar messages [ 3211.889373] Lustre: server umount lustre-OST0000 complete [ 3228.127263] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3228.211541] Lustre: server umount lustre-MDT0000 complete [ 3230.050981] LustreError: 217581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783152276 with bad export cookie 9619305391486942340 [ 3230.057842] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3230.058805] LustreError: 217581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3230.206208] Lustre: server umount lustre-MDT0001 complete [ 3234.394098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3234.636060] LustreError: 220969:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3234.677701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3234.682208] Lustre: Skipped 5 previous similar messages [ 3236.603011] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3240.630905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3242.537386] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3245.968378] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3248.524394] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3251.781088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3254.571969] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3258.038197] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3270.629285] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3276.647896] Lustre: server umount lustre-OST0000 complete [ 3280.866261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3280.875111] Lustre: Skipped 6 previous similar messages [ 3284.597824] Lustre: server umount lustre-MDT0000 complete [ 3286.387087] LustreError: 220951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783152332 with bad export cookie 9619305391486943222 [ 3286.393809] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3286.397861] LustreError: 220951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3286.538955] Lustre: server umount lustre-MDT0001 complete [ 3394.120198] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 3395.527411] Key type lgssc unregistered [ 3395.699548] LNet: 224650:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3395.704665] LNetError: 224650:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3395.718666] LNet: Removed LNI 192.168.204.146@tcp [ 3396.244192] Key type .llcrypt unregistered [ 3396.246845] Key type ._llcrypt unregistered [ 3407.982872] Key type ._llcrypt registered [ 3407.984534] Key type .llcrypt registered [ 3408.049576] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 3416.677185] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3417.061958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3417.113066] alg: No test for adler32 (adler32-zlib) [ 3418.004875] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 3418.137179] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 3419.751207] Key type lgssc registered [ 3420.388678] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3437.936372] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3437.945742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3439.093482] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3439.144661] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3439.203807] Lustre: lustre-MDT0000: new disk, initializing [ 3439.240544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3439.251941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3441.006738] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3446.683709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3446.759598] Lustre: 227976:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3446.809773] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3446.813834] Lustre: Skipped 1 previous similar message [ 3446.870468] Lustre: lustre-MDT0001: new disk, initializing [ 3446.925737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3446.947820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3446.955537] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3448.967616] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3454.004692] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3456.610578] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3460.041372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3460.227254] Lustre: lustre-OST0000: new disk, initializing [ 3460.230906] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3460.236387] Lustre: 229946:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3460.291324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3462.918096] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3467.659969] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3469.353854] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3470.345430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3470.351930] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3470.406879] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3470.498597] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3472.382314] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3472.500089] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3475.429861] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3475.430568] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3475.431515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3475.434248] LustreError: Skipped 1 previous similar message [ 3475.441594] Lustre: Skipped 1 previous similar message [ 3479.519104] Lustre: server umount lustre-OST0000 complete [ 3483.104589] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3483.109797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3483.118989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3483.123041] Lustre: Skipped 1 previous similar message [ 3485.665181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3485.666326] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3490.785064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3490.789642] Lustre: Skipped 2 previous similar messages [ 3495.391178] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3495.477971] Lustre: server umount lustre-MDT0000 complete [ 3497.274889] LustreError: 227968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783152543 with bad export cookie 8635318682165926213 [ 3497.282506] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3497.282722] LustreError: 227968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3497.433266] Lustre: server umount lustre-MDT0001 complete [ 3506.529671] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3511.298364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3511.462505] LustreError: 232882:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3511.469767] LustreError: 232882:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3511.477623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3513.495814] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3516.898341] LustreError: 232883:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3517.736427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3517.955559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3519.872722] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3521.294979] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3524.613280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3527.733326] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3531.932621] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3532.062959] Lustre: lustre-OST0001: new disk, initializing [ 3532.066738] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3532.071826] Lustre: 235192:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3532.135046] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3532.138157] Lustre: Skipped 1 previous similar message [ 3535.148683] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3535.597908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3535.604273] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3535.657910] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3541.867733] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3547.418895] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3553.608805] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 04:09:59 (1783152599) [ 3556.728174] Lustre: server umount lustre-MDT0000 complete [ 3557.347748] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3557.350504] LustreError: 232879:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3557.356049] Lustre: Skipped 2 previous similar messages [ 3557.369803] LustreError: 232879:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3558.512761] LustreError: 232863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783152604 with bad export cookie 8635318682165927081 [ 3558.518846] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3558.521089] LustreError: 232863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3558.673414] Lustre: server umount lustre-MDT0001 complete [ 3570.688595] Lustre: server umount lustre-OST0000 complete [ 3581.088675] Lustre: server umount lustre-OST0001 complete [ 3583.057319] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 3586.247245] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3597.384356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3597.465396] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3597.474589] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3597.510414] Lustre: lustre-MDT0000: new disk, initializing [ 3597.531334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3597.536791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3598.731431] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3602.650127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3602.689610] Lustre: 239142:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3602.799912] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3604.000589] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3607.407335] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3609.000946] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3611.418492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3611.588476] Lustre: lustre-OST0000: new disk, initializing [ 3611.591868] Lustre: Skipped 1 previous similar message [ 3611.595449] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3611.599572] Lustre: Skipped 2 previous similar messages [ 3611.603972] Lustre: 241108:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3613.131790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3613.137498] Lustre: Skipped 1 previous similar message [ 3613.141901] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3613.200570] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3614.182272] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3618.990889] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3620.938172] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3621.057500] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3623.058396] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3623.134807] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3628.516094] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3628.518430] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3628.519068] LustreError: Skipped 1 previous similar message [ 3628.522908] Lustre: Skipped 1 previous similar message [ 3628.524949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3628.527550] Lustre: Skipped 1 previous similar message [ 3638.239187] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3638.333407] Lustre: server umount lustre-OST0000 complete [ 3643.876299] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3643.876383] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3643.878044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3643.878057] Lustre: Skipped 2 previous similar messages [ 3643.884236] Lustre: Skipped 1 previous similar message [ 3646.393211] Lustre: server umount lustre-MDT0000 complete [ 3648.222345] LustreError: 239937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783152694 with bad export cookie 8635318682165929195 [ 3648.228819] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3648.229636] LustreError: 239937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3648.402789] Lustre: server umount lustre-MDT0001 complete [ 3652.621654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3652.866208] LustreError: 243324:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3652.910781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3652.916653] Lustre: Skipped 2 previous similar messages [ 3654.957900] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3658.209812] LustreError: 243325:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3659.033447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3660.883341] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3664.370902] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3667.023042] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3670.211619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3673.055461] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3676.186129] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3821.024799] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3821.027973] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3821.034545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3821.038323] Lustre: Skipped 1 previous similar message [ 3834.847125] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3834.924351] Lustre: server umount lustre-OST0000 complete [ 3838.435057] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3838.441307] LustreError: Skipped 1 previous similar message [ 3851.231174] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3851.318820] Lustre: server umount lustre-MDT0000 complete [ 3853.118114] LustreError: 243306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783152899 with bad export cookie 8635318682165930077 [ 3853.123883] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3853.125989] LustreError: 243306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3853.298552] Lustre: server umount lustre-MDT0001 complete [ 3942.339853] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 3943.937298] Key type lgssc unregistered [ 3944.122727] LNet: 246933:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3944.127701] LNetError: 246933:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3944.139700] LNet: Removed LNI 192.168.204.146@tcp [ 3944.659153] Key type .llcrypt unregistered [ 3944.661561] Key type ._llcrypt unregistered [ 3956.044138] Key type ._llcrypt registered [ 3956.046173] Key type .llcrypt registered [ 3956.104461] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 3963.992147] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3964.381020] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3964.420985] alg: No test for adler32 (adler32-zlib) [ 3965.289213] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 3965.405226] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 3967.007233] Key type lgssc registered [ 3967.655651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3989.156759] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3993.821784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3993.830079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3994.966771] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3994.981554] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3995.040036] Lustre: lustre-MDT0000: new disk, initializing [ 3995.076372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3995.087807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3997.057315] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4002.570938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4002.647693] Lustre: 250973:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4002.693247] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4002.696966] Lustre: Skipped 1 previous similar message [ 4002.749227] Lustre: lustre-MDT0001: new disk, initializing [ 4002.801232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4002.823532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4002.829824] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4004.577183] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4007.401761] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4010.619126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4010.816837] Lustre: lustre-OST0000: new disk, initializing [ 4010.820428] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4010.825533] Lustre: 252606:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4010.889734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4013.642291] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4014.553107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4014.561173] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4014.638371] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4019.176482] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4019.294683] Lustre: lustre-OST0001: new disk, initializing [ 4019.297309] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4019.301184] Lustre: 253477:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4019.365910] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4022.158626] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4023.846774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4023.854035] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4023.917254] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4028.694743] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4032.055892] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4038.624609] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4038.629674] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4038.638677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4039.139920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4039.143335] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4039.153324] Lustre: Skipped 2 previous similar messages [ 4042.631981] Lustre: server umount lustre-MDT0000 complete [ 4044.412059] LustreError: 250966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783153090 with bad export cookie 3085741224627538544 [ 4044.419299] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4044.419834] LustreError: 250966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4044.605735] Lustre: server umount lustre-MDT0001 complete [ 4056.600642] Lustre: server umount lustre-OST0000 complete [ 4068.742508] Lustre: server umount lustre-OST0001 complete [ 4075.410802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4079.955704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4083.993575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4088.547341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4097.546329] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4102.570903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4102.585340] Lustre: lustre-MDT0000: reset Object Index mappings [ 4102.821747] LustreError: 258478:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4102.831491] LustreError: 258478:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4102.847199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4104.853901] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4108.256942] LustreError: 258497:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4109.089937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4109.102321] Lustre: lustre-MDT0001: reset Object Index mappings [ 4111.430663] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4112.927569] Lustre: 259623:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4116.178639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4116.402829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4116.407279] Lustre: Skipped 1 previous similar message [ 4119.357835] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4119.468111] LustreError: 259975:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4123.589816] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4126.634446] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4127.857587] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4127.878862] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 4136.175378] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4138.025103] Lustre: 261489:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4140.001767] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4140.006883] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4140.015741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4140.019892] Lustre: Skipped 3 previous similar messages [ 4143.594417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4143.594894] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4143.600451] Lustre: Skipped 2 previous similar messages [ 4143.608541] Lustre: Skipped 2 previous similar messages [ 4145.867187] Lustre: server umount lustre-MDT0000 complete [ 4147.645467] LustreError: 258461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783153193 with bad export cookie 3085741224627545320 [ 4147.650312] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4147.653112] LustreError: 258461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4147.853561] Lustre: server umount lustre-MDT0001 complete [ 4158.997926] Lustre: server umount lustre-OST0000 complete [ 4170.049150] Lustre: server umount lustre-OST0001 complete [ 4179.339650] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 04:20:24 (1783153224) [ 4186.904844] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 4191.748146] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4209.592306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4209.744703] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4209.763429] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4209.819225] Lustre: lustre-MDT0000: new disk, initializing [ 4209.854544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4209.859087] Lustre: Skipped 1 previous similar message [ 4209.868924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4211.752316] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4217.239379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4217.304690] Lustre: 265740:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4217.449914] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4219.272359] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4224.227337] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4226.895170] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4230.213824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4230.390711] Lustre: lustre-OST0000: new disk, initializing [ 4230.393757] Lustre: Skipped 1 previous similar message [ 4230.397107] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4230.401515] Lustre: Skipped 2 previous similar messages [ 4230.405552] Lustre: 267708:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4231.818852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4231.823311] Lustre: Skipped 1 previous similar message [ 4231.826388] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4231.863799] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4233.010541] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4237.957736] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4239.921797] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4240.041355] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4241.976602] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4242.097315] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4247.524352] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4247.526105] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4247.527824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4247.529435] LustreError: Skipped 1 previous similar message [ 4247.537438] Lustre: Skipped 1 previous similar message [ 4257.247138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4257.327444] Lustre: server umount lustre-OST0000 complete [ 4262.882380] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4262.883366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4262.890413] Lustre: Skipped 1 previous similar message [ 4262.894427] Lustre: Skipped 3 previous similar messages [ 4265.334198] Lustre: server umount lustre-MDT0000 complete [ 4267.126940] LustreError: 266533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783153313 with bad export cookie 3085741224627547392 [ 4267.132300] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4267.134414] LustreError: 266533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4267.280618] Lustre: server umount lustre-MDT0001 complete [ 4275.958906] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4280.792723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4280.995589] LustreError: 270638:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4281.017164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4281.021348] Lustre: Skipped 2 previous similar messages [ 4282.810186] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4286.432968] LustreError: 270638:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4286.443519] LustreError: 270638:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4286.916179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4288.908132] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4290.303038] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4293.460785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4296.335365] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4300.454359] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4300.567065] Lustre: lustre-OST0001: new disk, initializing [ 4300.570415] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4300.575624] Lustre: 272944:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4302.184207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4302.190741] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4302.251042] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4303.487137] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4309.215554] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4311.004802] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4313.569431] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 4313.578973] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 4316.645827] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4316.653255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4316.653730] Lustre: Skipped 3 previous similar messages [ 4316.657555] Lustre: Skipped 3 previous similar messages [ 4322.444502] Lustre: server umount lustre-MDT0000 complete [ 4324.249855] LustreError: 270620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783153370 with bad export cookie 3085741224627548274 [ 4324.257247] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4324.257733] LustreError: 270620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4324.405668] Lustre: server umount lustre-MDT0001 complete [ 4336.091942] Lustre: server umount lustre-OST0000 complete [ 4347.132549] Lustre: server umount lustre-OST0001 complete [ 4354.902952] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4359.697648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4359.922785] LustreError: 275995:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4359.951582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4359.955546] Lustre: Skipped 3 previous similar messages [ 4361.889790] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4365.993936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4368.015007] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4369.427338] Lustre: 277136:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4372.571637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4375.472469] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4376.874874] LustreError: 277490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4376.886985] LustreError: 277490:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4379.591311] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4382.480561] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4385.787432] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 4385.793126] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 4392.140086] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4393.921831] Lustre: 279016:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4396.515946] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 4396.524090] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 4401.184845] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 04:24:06 (1783153446) [ 4414.438757] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4414.441188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4414.446666] Lustre: Skipped 2 previous similar messages [ 4414.450599] Lustre: Skipped 5 previous similar messages [ 4415.154459] Lustre: server umount lustre-MDT0000 complete [ 4416.895194] LustreError: 275976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783153462 with bad export cookie 3085741224627554588 [ 4416.902072] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4416.902681] LustreError: 275976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4419.552987] LustreError: 275992:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4419.564038] LustreError: 275992:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4423.131931] Lustre: server umount lustre-MDT0001 complete [ 4431.049874] Lustre: server umount lustre-OST0000 complete [ 4447.199199] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4447.292288] Lustre: server umount lustre-OST0001 complete [ 4450.309661] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 4454.980370] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4477.330736] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4481.979432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4482.116111] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4482.132433] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4482.178659] Lustre: lustre-MDT0000: new disk, initializing [ 4482.210519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4484.129185] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4489.745081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4489.816281] Lustre: 283250:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4489.953293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4489.956927] Lustre: Skipped 4 previous similar messages [ 4489.976115] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4491.814193] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4497.812870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4497.975364] Lustre: 284885:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4499.901769] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4500.639275] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4506.188583] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4506.296342] Lustre: 285756:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4507.859988] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4509.114068] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4514.785906] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4516.553850] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4516.557958] Lustre: Skipped 1 previous similar message [ 4522.760579] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 04:26:08 (1783153568) [ 4526.048362] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4526.053481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4526.061420] Lustre: Skipped 3 previous similar messages [ 4526.065115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4526.069102] Lustre: Skipped 6 previous similar messages [ 4529.918843] Lustre: server umount lustre-MDT0000 complete [ 4531.659881] LustreError: 283242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783153577 with bad export cookie 3085741224627562449 [ 4531.665271] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4531.667445] LustreError: 283242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4531.839238] Lustre: server umount lustre-MDT0001 complete [ 4543.951738] Lustre: server umount lustre-OST0000 complete [ 4556.027442] Lustre: server umount lustre-OST0001 complete [ 4560.163396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4560.347064] LustreError: 287998:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4560.353271] LustreError: 287998:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4562.166100] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4566.241359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4568.235468] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4571.724006] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4574.456431] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4577.343902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4579.405919] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4581.984152] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4596.831035] loop1: detected capacity change from 0 to 204800000 [ 4596.910080] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4596.913885] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4596.917123] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4598.359089] loop1: detected capacity change from 0 to 204800000 [ 4600.782690] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4600.828243] LustreError: 287988:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs32722: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 4600.835326] LustreError: 287988:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs32722 [ 4600.838657] LustreError: 287988:0:(mgs_handler.c:517:mgs_target_reg()) fs32722-MDT0001: Error checking target index: rc = -13 [ 4600.843450] LustreError: fs32722-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 4600.846515] LustreError: 291731:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 4600.848868] LustreError: 291731:0:(tgt_mount.c:2129:server_put_super()) no obd fs32722-MDT0001 [ 4600.850935] LustreError: 291731:0:(tgt_mount.c:116:server_deregister_mount()) fs32722-MDT0001 not registered [ 4600.864340] Lustre: server umount fs32722-MDT0001 complete [ 4600.866775] LustreError: 291731:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 4604.897827] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4604.948457] Lustre: Setting parameter fs32722-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs32722-MDT0001 [ 4604.978914] Lustre: srv-fs32722-MDT0001: No data found on store. Initialize space. [ 4604.982985] Lustre: Skipped 4 previous similar messages [ 4605.035204] Lustre: fs32722-MDT0001: new disk, initializing [ 4605.036719] Lustre: Skipped 3 previous similar messages [ 4605.074814] Lustre: srv-fs32722-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 4605.077267] Lustre: 292197:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs32722-MDT0001-mdtlov: can't create new llog: rc = -115 [ 4605.081302] LustreError: 292197:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs32722-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 4606.792004] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4610.515919] Lustre: server umount fs32722-MDT0001 complete [ 4616.681827] loop1: detected capacity change from 0 to 204800000 [ 4618.884937] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4619.038434] Lustre: 293763:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs32722-MDT0001-mdtlov: can't create new llog: rc = -115 [ 4619.042279] LustreError: 293763:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs32722-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 4620.687712] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4622.770181] Lustre: server umount fs32722-MDT0001 complete [ 4627.939369] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4641.247210] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4641.344894] Lustre: server umount lustre-MDT0000 complete [ 4645.325057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4645.405140] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4647.287675] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4650.938385] loop1: detected capacity change from 0 to 204800000 [ 4650.994059] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4653.137400] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4653.245073] Lustre: 296207:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs32722-MDT0001-mdtlov: can't create new llog: rc = -115 [ 4653.248192] LustreError: 296207:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs32722-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 4654.906671] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4656.669807] Lustre: server umount fs32722-MDT0001 complete [ 4663.775674] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4663.777880] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4663.781090] Lustre: Skipped 6 previous similar messages [ 4663.782601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4663.784204] Lustre: Skipped 16 previous similar messages [ 4677.087176] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4677.167260] Lustre: server umount lustre-OST0000 complete [ 4681.699103] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4681.704018] LustreError: Skipped 2 previous similar messages [ 4685.039292] Lustre: server umount lustre-MDT0000 complete [ 4686.706959] LustreError: 296698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783153732 with bad export cookie 3085741224627565130 [ 4686.713171] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4686.714272] LustreError: 296698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4686.859604] Lustre: server umount lustre-MDT0001 complete [ 4691.357797] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 4692.855341] Key type lgssc unregistered [ 4693.027663] LNet: 298547:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4693.032525] LNetError: 298547:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4693.043561] LNet: Removed LNI 192.168.204.146@tcp [ 4693.508185] Key type .llcrypt unregistered [ 4693.510444] Key type ._llcrypt unregistered [ 4702.718530] Key type ._llcrypt registered [ 4702.719672] Key type .llcrypt registered [ 4702.761862] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4703.157630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4703.177968] alg: No test for adler32 (adler32-zlib) [ 4704.078402] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 4704.208238] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 4705.807175] Key type lgssc registered [ 4706.311263] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4709.773144] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4709.784120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4710.983182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4712.770617] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4716.610161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4716.794175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4718.347209] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4721.564348] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4724.015835] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4727.167163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4727.367934] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4729.951818] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4733.252293] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4736.614675] LustreError: 301624:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4736.624877] LustreError: 301624:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4739.692818] LustreError: 301625:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4741.734245] LustreError: 301624:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4741.745086] LustreError: 301624:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4744.828354] LustreError: 301623:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4749.948827] LustreError: 302294:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4749.959390] LustreError: 302294:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4751.491427] loop1: detected capacity change from 0 to 204800000 [ 4751.584147] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4751.589386] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4751.595693] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4753.971710] loop1: detected capacity change from 0 to 204800000 [ 4756.535083] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4756.610887] Lustre: fs10883-OST0000: new disk, initializing [ 4756.614560] Lustre: srv-fs10883-OST0000: No data found on store. Initialize space. [ 4756.619913] Lustre: 303551:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4756.660242] Lustre: fs10883-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4759.276459] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4760.188159] LustreError: 301623:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4760.198853] LustreError: 301623:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4763.746422] Lustre: server umount fs10883-OST0000 complete [ 4768.514976] loop1: detected capacity change from 0 to 204800000 [ 4770.118753] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4770.171957] Lustre: fs10883-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4771.766087] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4774.179789] loop2: detected capacity change from 0 to 204800000 [ 4774.261634] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4774.265148] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4774.268657] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4775.713815] loop2: detected capacity change from 0 to 204800000 [ 4777.319193] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4777.345383] LustreError: fs10883-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 4777.348977] LustreError: fs10883-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 4777.352152] LustreError: 306586:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 4777.354365] LustreError: 306586:0:(tgt_mount.c:2129:server_put_super()) no obd fs10883-OST0001 [ 4777.356345] LustreError: 306586:0:(tgt_mount.c:116:server_deregister_mount()) fs10883-OST0001 not registered [ 4777.364666] Lustre: server umount fs10883-OST0001 complete [ 4777.365962] LustreError: 306586:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 4777.953900] LustreError: 301625:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4777.958627] LustreError: 301625:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 4781.711442] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4781.795080] Lustre: fs10883-OST0001: new disk, initializing [ 4781.798627] Lustre: srv-fs10883-OST0001: No data found on store. Initialize space. [ 4781.804241] Lustre: 307137:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4781.845504] Lustre: fs10883-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4784.318844] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4788.461960] Lustre: server umount fs10883-OST0000 complete [ 4792.926858] Lustre: server umount fs10883-OST0001 complete [ 4798.754798] loop1: detected capacity change from 0 to 204800000 [ 4801.465704] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4801.576742] Lustre: fs10883-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4804.157770] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4807.341633] loop2: detected capacity change from 0 to 204800000 [ 4810.092412] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4811.387388] LustreError: 302295:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4811.398844] LustreError: 302295:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [ 4813.012776] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4817.365681] Lustre: server umount fs10883-OST0000 complete [ 4821.756674] Lustre: server umount fs10883-OST0001 complete [ 4824.046568] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [ 4829.154254] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4829.159534] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4829.162824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4829.167404] Lustre: Skipped 1 previous similar message [ 4833.259662] Lustre: server umount lustre-OST0000 complete [ 4839.394347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4839.395312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4839.402448] Lustre: Skipped 1 previous similar message [ 4839.409159] Lustre: Skipped 2 previous similar messages [ 4841.272545] Lustre: server umount lustre-MDT0000 complete [ 4842.931264] LustreError: 307136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783153889 with bad export cookie 6073676733751909760 [ 4842.934422] LustreError: 307136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4842.934694] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4843.052447] Lustre: server umount lustre-MDT0001 complete [ 4847.836206] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 4849.327236] Key type lgssc unregistered [ 4849.482681] LNet: 313052:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4849.487844] LNetError: 313052:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4849.497562] LNet: Removed LNI 192.168.204.146@tcp [ 4849.966173] Key type .llcrypt unregistered [ 4849.968518] Key type ._llcrypt unregistered [ 4864.591695] Key type ._llcrypt registered [ 4864.593083] Key type .llcrypt registered [ 4864.637746] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 4873.347938] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4873.651891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4873.683051] alg: No test for adler32 (adler32-zlib) [ 4874.583586] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 4874.709470] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 4876.319160] Key type lgssc registered [ 4876.932384] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4895.207399] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4895.217325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4896.321447] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4896.334438] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4896.379795] Lustre: lustre-MDT0000: new disk, initializing [ 4896.408302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4896.418545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4898.082448] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4903.677470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4903.747045] Lustre: 316884:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4903.785465] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4903.789190] Lustre: Skipped 1 previous similar message [ 4903.836622] Lustre: lustre-MDT0001: new disk, initializing [ 4903.872767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4903.884675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4903.887835] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4905.539156] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4909.274477] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4910.969994] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4913.143390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4913.265777] Lustre: lustre-OST0000: new disk, initializing [ 4913.268588] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4913.272191] Lustre: 318851:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4913.306663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4915.375953] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4916.557288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4916.562244] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4916.607150] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4919.686876] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4920.546816] Lustre: Mounted lustre-client [ 4920.957588] Lustre: Unmounted lustre-client [ 4925.922123] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4925.933025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4925.940433] Lustre: Skipped 1 previous similar message [ 4926.829995] Lustre: server umount lustre-OST0000 complete [ 4929.504270] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4929.507667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4929.513380] Lustre: Skipped 1 previous similar message [ 4929.516455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4931.040815] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4931.041383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4931.050503] Lustre: Skipped 1 previous similar message [ 4934.501277] Lustre: server umount lustre-MDT0000 complete [ 4936.068872] LustreError: 316875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783153982 with bad export cookie 14802076476216553378 [ 4936.072830] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4936.077333] LustreError: 316875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4936.236195] Lustre: server umount lustre-MDT0001 complete [ 4940.773525] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 4942.189511] Key type lgssc unregistered [ 4979.231394] LNet: 321010:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4979.235608] LNetError: 321010:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5051.941426] LNet: Removed LNI 192.168.204.146@tcp [ 5052.274349] Key type .llcrypt unregistered [ 5052.275757] Key type ._llcrypt unregistered [ 5059.021512] Key type ._llcrypt registered [ 5059.022574] Key type .llcrypt registered [ 5059.068202] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 5066.342228] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 5066.898525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5066.908763] alg: No test for adler32 (adler32-zlib) [ 5067.772856] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 5067.873876] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 5069.455150] Key type lgssc registered [ 5069.880664] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5082.300313] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 04:35:27 (1783154127) [ 5084.865349] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5084.873767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5085.970750] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5085.983415] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5086.020509] Lustre: lustre-MDT0000: new disk, initializing [ 5086.052804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5086.062501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5087.675534] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5092.239874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5092.293791] Lustre: 324221:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5092.321954] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5092.324637] Lustre: Skipped 1 previous similar message [ 5092.364514] Lustre: lustre-MDT0001: new disk, initializing [ 5092.405876] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5092.420897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5092.425376] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5094.018063] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5098.206326] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5100.393480] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5103.060898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5103.175805] Lustre: lustre-OST0000: new disk, initializing [ 5103.179134] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5103.183985] Lustre: 326189:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5103.222409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5105.382947] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5109.175751] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5110.778455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5110.782673] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5110.821859] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5120.993105] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5120.994769] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5120.998514] LustreError: Skipped 1 previous similar message [ 5121.002990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5121.005809] Lustre: Skipped 1 previous similar message [ 5125.902251] Lustre: server umount lustre-OST0000 complete [ 5128.159893] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5128.162156] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5128.166365] Lustre: Skipped 1 previous similar message [ 5128.168299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5131.232161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5131.232709] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5133.216120] Lustre: server umount lustre-MDT0000 complete [ 5134.456493] LustreError: 327060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783154180 with bad export cookie 10268373917330118400 [ 5134.459106] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5134.461188] LustreError: 327060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5134.573545] Lustre: server umount lustre-MDT0001 complete [ 5138.340660] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 5139.395503] Key type lgssc unregistered [ 5139.523371] LNet: 328101:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5139.525956] LNetError: 328101:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5139.537386] LNet: Removed LNI 192.168.204.146@tcp [ 5139.862250] Key type .llcrypt unregistered [ 5139.863629] Key type ._llcrypt unregistered [ 5148.085842] Key type ._llcrypt registered [ 5148.087327] Key type .llcrypt registered [ 5148.125391] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 5148.606444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5148.622789] alg: No test for adler32 (adler32-zlib) [ 5149.477818] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 5149.569790] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 5151.151125] Key type lgssc registered [ 5151.521839] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5154.189979] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5154.196046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5155.340516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5156.715724] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5159.561175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5159.687869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5161.120808] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5163.595502] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5165.434296] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5167.758233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5167.871921] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5169.686334] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5172.152882] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5186.149480] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5186.738224] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5190.624080] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5190.627708] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5190.633966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5191.648741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5191.649315] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5191.650650] Lustre: Skipped 1 previous similar message [ 5191.654045] Lustre: Skipped 1 previous similar message [ 5192.830848] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.46@tcp (stopping) [ 5192.832851] Lustre: Skipped 1 previous similar message [ 5194.197084] Lustre: server umount lustre-MDT0000 complete [ 5195.423662] LustreError: 331192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783154241 with bad export cookie 2838208392550192072 [ 5195.426818] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5195.428566] LustreError: 331192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5196.767861] LustreError: 329346:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5196.768174] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5196.774774] LustreError: 329346:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5196.779103] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5197.944375] LustreError: 329348:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5200.799807] LustreError: 329346:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5200.800081] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5200.806306] Lustre: Skipped 1 previous similar message [ 5201.579052] Lustre: server umount lustre-MDT0001 complete [ 5213.304154] Lustre: lustre-OST0000: Not available for connect from 192.168.204.46@tcp (stopping) [ 5213.306755] Lustre: Skipped 2 previous similar messages [ 5217.247174] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5217.313327] Lustre: server umount lustre-OST0000 complete [ 5220.214432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5221.506344] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5222.409694] Lustre: 333545:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.146@tcp [ 5223.105542] Lustre: server umount MGS complete [ 5225.768037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5225.911342] LustreError: 334117:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5225.933118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5227.246783] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5229.974791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5230.093336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5231.450224] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5234.242570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5236.372425] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5238.788735] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5244.474930] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec [ 5246.336483] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5248.937875] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5251.043224] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5251.045494] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5251.046389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5251.046393] Lustre: Skipped 1 previous similar message [ 5251.047363] Lustre: Skipped 1 previous similar message [ 5251.050289] LustreError: Skipped 1 previous similar message [ 5256.267116] Lustre: server umount lustre-OST0000 complete [ 5260.768310] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5260.771314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5263.643343] Lustre: server umount lustre-MDT0000 complete [ 5264.933240] LustreError: 334098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783154311 with bad export cookie 2838208392550192898 [ 5264.936480] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5264.937782] LustreError: 334098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5265.056451] Lustre: server umount lustre-MDT0001 complete [ 5268.827812] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 5269.818835] Key type lgssc unregistered [ 5269.943377] LNet: 337550:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5269.945798] LNetError: 337550:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5269.960301] LNet: Removed LNI 192.168.204.146@tcp [ 5270.264971] Key type .llcrypt unregistered [ 5270.266171] Key type ._llcrypt unregistered [ 5276.337590] Key type ._llcrypt registered [ 5276.338865] Key type .llcrypt registered [ 5276.386446] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 5283.135963] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 5283.627201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5283.662830] alg: No test for adler32 (adler32-zlib) [ 5284.513971] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 5284.607147] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 5286.183194] Key type lgssc registered [ 5286.625774] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5299.086654] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5299.092230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5300.174431] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5300.186659] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5300.225653] Lustre: lustre-MDT0000: new disk, initializing [ 5300.250795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5300.258720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5301.768395] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5306.163445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5306.212609] Lustre: 340681:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5306.241350] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5306.243542] Lustre: Skipped 1 previous similar message [ 5306.281280] Lustre: lustre-MDT0001: new disk, initializing [ 5306.325461] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5306.340509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5306.344910] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5307.775374] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5311.734941] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5313.671759] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5316.084229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5316.192333] Lustre: lustre-OST0000: new disk, initializing [ 5316.193948] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5316.196760] Lustre: 342646:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5316.226268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5317.622864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5317.627638] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5317.656287] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5318.230338] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5321.879788] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5323.272941] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5323.340525] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5324.636866] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5324.707221] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5327.841154] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5327.842179] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5327.844447] LustreError: Skipped 1 previous similar message [ 5327.849616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5327.853686] Lustre: Skipped 1 previous similar message [ 5331.407233] Lustre: server umount lustre-OST0000 complete [ 5332.960159] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5332.960719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5332.965040] Lustre: Skipped 2 previous similar messages [ 5338.080302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5338.082746] Lustre: Skipped 2 previous similar messages [ 5338.707241] Lustre: server umount lustre-MDT0000 complete [ 5339.869135] LustreError: 341476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783154385 with bad export cookie 8214140614588595015 [ 5339.871697] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5339.873270] LustreError: 341476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5339.990625] Lustre: server umount lustre-MDT0001 complete [ 5342.620995] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 04:39:48 (1783154388) [ 5347.518357] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 5350.561079] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 5361.370016] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5361.446709] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5361.457147] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5361.489363] Lustre: lustre-MDT0000: new disk, initializing [ 5361.507038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5361.512442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5362.697539] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5366.550429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5366.601061] Lustre: 347395:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5366.626044] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5366.628161] Lustre: Skipped 1 previous similar message [ 5366.661064] Lustre: lustre-MDT0001: new disk, initializing [ 5366.690724] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5366.703017] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5366.707901] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5368.027444] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5371.481505] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5373.171974] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5375.327289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5375.426832] Lustre: lustre-OST0000: new disk, initializing [ 5375.428911] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5375.431944] Lustre: 349363:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5375.456332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5377.332321] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5377.909694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5377.912864] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5377.946734] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5380.918263] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5393.376615] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5393.378961] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5393.383798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5393.386261] Lustre: Skipped 1 previous similar message [ 5396.053341] Lustre: server umount lustre-OST0000 complete [ 5397.471862] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5397.473990] LustreError: Skipped 1 previous similar message [ 5397.475437] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5397.478688] Lustre: Skipped 1 previous similar message [ 5397.480346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5403.410222] Lustre: server umount lustre-MDT0000 complete [ 5404.588069] LustreError: 348190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783154450 with bad export cookie 8214140614588595883 [ 5404.591171] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5404.592589] LustreError: 348190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5404.696156] Lustre: server umount lustre-MDT0001 complete [ 5408.297565] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 5409.299331] Key type lgssc unregistered [ 5409.431432] LNet: 351507:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5409.433979] LNetError: 351507:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5409.442378] LNet: Removed LNI 192.168.204.146@tcp [ 5409.729622] Key type .llcrypt unregistered [ 5409.730717] Key type ._llcrypt unregistered [ 5440.735553] Key type ._llcrypt registered [ 5440.737149] Key type .llcrypt registered [ 5440.774640] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 5440.878204] Key type .llcrypt unregistered [ 5440.879528] Key type ._llcrypt unregistered [ 5466.540382] Key type ._llcrypt registered [ 5466.541755] Key type .llcrypt registered [ 5466.577531] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 5466.691421] Key type .llcrypt unregistered [ 5466.693043] Key type ._llcrypt unregistered [ 5477.207945] Key type ._llcrypt registered [ 5477.208902] Key type .llcrypt registered [ 5477.243021] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 5477.337216] Key type .llcrypt unregistered [ 5477.338287] Key type ._llcrypt unregistered [ 5487.948274] Key type ._llcrypt registered [ 5487.949310] Key type .llcrypt registered [ 5487.991408] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 5488.083220] Key type .llcrypt unregistered [ 5488.085285] Key type ._llcrypt unregistered [ 5514.740317] Key type ._llcrypt registered [ 5514.741250] Key type .llcrypt registered [ 5514.783489] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 5514.898335] Key type .llcrypt unregistered [ 5514.899652] Key type ._llcrypt unregistered [ 5525.318035] Key type ._llcrypt registered [ 5525.319199] Key type .llcrypt registered [ 5525.353496] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 5525.442438] Key type .llcrypt unregistered [ 5525.443553] Key type ._llcrypt unregistered [ 5535.949163] Key type ._llcrypt registered [ 5535.950700] Key type .llcrypt registered [ 5535.985810] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 5536.092266] Key type .llcrypt unregistered [ 5536.093416] Key type ._llcrypt unregistered [ 5546.965376] Key type ._llcrypt registered [ 5546.966302] Key type .llcrypt registered [ 5547.002056] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 5547.106384] Key type .llcrypt unregistered [ 5547.107594] Key type ._llcrypt unregistered [ 5557.696943] Key type ._llcrypt registered [ 5557.698197] Key type .llcrypt registered [ 5557.733402] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 5557.825218] Key type .llcrypt unregistered [ 5557.826108] Key type ._llcrypt unregistered [ 5568.574530] Key type ._llcrypt registered [ 5568.575660] Key type .llcrypt registered [ 5568.615211] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 5568.721318] Key type .llcrypt unregistered [ 5568.722958] Key type ._llcrypt unregistered [ 5588.727835] Key type ._llcrypt registered [ 5588.728946] Key type .llcrypt registered [ 5588.768544] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 5588.873235] Key type .llcrypt unregistered [ 5588.874615] Key type ._llcrypt unregistered [ 5599.394781] Key type ._llcrypt registered [ 5599.396044] Key type .llcrypt registered [ 5599.433819] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 5599.537194] Key type .llcrypt unregistered [ 5599.538921] Key type ._llcrypt unregistered [ 5610.076643] Key type ._llcrypt registered [ 5610.077736] Key type .llcrypt registered [ 5610.113651] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 5610.206363] Key type .llcrypt unregistered [ 5610.208039] Key type ._llcrypt unregistered [ 5628.104591] Key type ._llcrypt registered [ 5628.105798] Key type .llcrypt registered [ 5628.143898] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 5634.478643] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 5634.834462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5634.866238] alg: No test for adler32 (adler32-zlib) [ 5635.741893] Lustre: Lustre: Build Version: 2.17.54_84_g3203d2f [ 5635.847311] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 5637.439174] Key type lgssc registered [ 5637.878283] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5649.512170] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5649.518164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5650.620789] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 5650.631435] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 5650.672208] Lustre: tst250fs-MDT0000: new disk, initializing [ 5650.696700] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5650.703928] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5652.127487] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5656.255797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5656.303509] Lustre: 371408:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5656.330736] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 5656.332864] Lustre: Skipped 1 previous similar message [ 5656.370763] Lustre: tst250fs-MDT0001: new disk, initializing [ 5656.402293] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5656.415073] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5656.418277] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5657.818610] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5660.109084] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5662.324029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5662.452917] Lustre: tst250fs-OST0000: new disk, initializing [ 5662.454840] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 5662.457386] Lustre: 373042:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5662.493655] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5664.496569] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5667.320879] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5667.324996] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5667.363214] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5668.729341] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5668.789398] Lustre: tst250fs-OST0001: new disk, initializing [ 5668.791353] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 5668.794436] Lustre: 373909:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5668.831325] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5670.800117] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5673.330888] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 5675.313704] Lustre: 371412:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.204.146@tcp: IR log tst250fs-mdtir failed, not fatal: rc = -2 [ 5684.194627] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5684.196047] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 5684.199754] Lustre: Skipped 3 previous similar messages [ 5684.202188] Lustre: Skipped 2 previous similar messages [ 5689.312263] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 5689.315403] Lustre: Skipped 2 previous similar messages [ 5689.566440] Lustre: server umount tst250fs-MDT0000 complete [ 5690.728220] LustreError: 372202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783154736 with bad export cookie 15282539304328336432 [ 5690.731486] LustreError: 372202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5690.732037] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5690.842752] Lustre: server umount tst250fs-MDT0001 complete [ 5702.663127] Lustre: server umount tst250fs-OST0000 complete [ 5714.311670] Lustre: server umount tst250fs-OST0001 complete [ 5725.527537] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 5733.230590] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 5757.455478] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 04:46:43 (1783154803) [ 5758.146197] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 5765.587233] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 5769.836065] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 5782.413405] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5668 sec ========== 04:47:08 (1783154828) [ 5783.137627] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 04:47:08 (1783154828) === [ 5784.593233] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 04:47:10 (1783154830) === [ 5801.600920] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 5803.017344] Key type lgssc unregistered [ 5803.181602] LNet: 383016:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5803.184744] LNetError: 383016:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5803.196241] LNet: Removed LNI 192.168.204.146@tcp [ 5803.578571] Key type .llcrypt unregistered [ 5803.582424] Key type ._llcrypt unregistered