[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 508479039 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.003267] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.006018] kvm-guest: setup PV IPIs [ 0.009000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.009026] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011143] LSM: Security Framework initializing [ 0.012063] Yama: becoming mindful. [ 0.013045] SELinux: Initializing. [ 0.015011] *** VALIDATE selinux *** [ 0.023817] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027869] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028176] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029125] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030117] *** VALIDATE tmpfs *** [ 0.031513] *** VALIDATE proc *** [ 0.032259] *** VALIDATE cgroup *** [ 0.033012] *** VALIDATE cgroup2 *** [ 0.034292] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035160] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037031] Spectre V2 : User space: Vulnerable [ 0.038013] Speculative Store Bypass: Vulnerable [ 0.041531] debug: unmapping init [mem 0xffffffff96459000-0xffffffff96460fff] [ 0.044000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044715] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045026] ... version: 2 [ 0.046015] ... bit width: 48 [ 0.047016] ... generic registers: 4 [ 0.048018] ... value mask: 0000ffffffffffff [ 0.049017] ... max period: 00007fffffffffff [ 0.050020] ... fixed-purpose events: 3 [ 0.051015] ... event mask: 000000070000000f [ 0.052330] rcu: Hierarchical SRCU implementation. [ 0.054513] smp: Bringing up secondary CPUs ... [ 0.055610] x86: Booting SMP configuration: [ 0.056028] .... node #0, CPUs: #1 #2 #3 [ 0.062648] smp: Brought up 1 node, 4 CPUs [ 0.064014] smpboot: Max logical packages: 1 [ 0.065024] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.139648] node 0 deferred pages initialised in 71ms [ 0.142197] devtmpfs: initialized [ 0.143257] x86/mm: Memory block size: 128MB [ 0.145878] gcov: version magic: 0x41383552 [ 0.146636] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.151084] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.152280] pinctrl core: initialized pinctrl subsystem [ 0.153211] [ 0.153810] ************************************************************* [ 0.154017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.155017] ** ** [ 0.156013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.157016] ** ** [ 0.158015] ** This means that this kernel is built to expose internal ** [ 0.159014] ** IOMMU data structures, which may compromise security on ** [ 0.160014] ** your system. ** [ 0.161018] ** ** [ 0.162020] ** If you see this message and you are not debugging the ** [ 0.163013] ** kernel, report this immediately to your vendor! ** [ 0.164016] ** ** [ 0.165015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166017] ************************************************************* [ 0.167667] NET: Registered protocol family 16 [ 0.168462] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.169064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.170062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.171437] cpuidle: using governor menu [ 0.173641] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.176648] PCI: Using configuration type 1 for base access [ 0.179235] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.189135] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.192074] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.195084] cryptd: max_cpu_qlen set to 1000 [ 0.197391] ACPI: Added _OSI(Module Device) [ 0.199012] ACPI: Added _OSI(Processor Device) [ 0.200014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.202016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.208185] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.216380] ACPI: Interpreter enabled [ 0.218063] ACPI: PM: (supports S0 S3 S4 S5) [ 0.219012] ACPI: Using IOAPIC for interrupt routing [ 0.221115] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.227399] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.237389] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.239046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.242025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.245086] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.250455] acpiphp: Slot [2] registered [ 0.252185] acpiphp: Slot [5] registered [ 0.254159] acpiphp: Slot [6] registered [ 0.255156] acpiphp: Slot [7] registered [ 0.257195] acpiphp: Slot [8] registered [ 0.259120] acpiphp: Slot [9] registered [ 0.260164] acpiphp: Slot [10] registered [ 0.262090] acpiphp: Slot [3] registered [ 0.263000] acpiphp: Slot [4] registered [ 0.263000] acpiphp: Slot [11] registered [ 0.264165] acpiphp: Slot [12] registered [ 0.266110] acpiphp: Slot [13] registered [ 0.267188] acpiphp: Slot [14] registered [ 0.269124] acpiphp: Slot [15] registered [ 0.271117] acpiphp: Slot [16] registered [ 0.272107] acpiphp: Slot [17] registered [ 0.274116] acpiphp: Slot [18] registered [ 0.275196] acpiphp: Slot [19] registered [ 0.277141] acpiphp: Slot [20] registered [ 0.279136] acpiphp: Slot [21] registered [ 0.280118] acpiphp: Slot [22] registered [ 0.282115] acpiphp: Slot [23] registered [ 0.284111] acpiphp: Slot [24] registered [ 0.285133] acpiphp: Slot [25] registered [ 0.287122] acpiphp: Slot [26] registered [ 0.289127] acpiphp: Slot [27] registered [ 0.290171] acpiphp: Slot [28] registered [ 0.292132] acpiphp: Slot [29] registered [ 0.294081] acpiphp: Slot [30] registered [ 0.295115] acpiphp: Slot [31] registered [ 0.297067] PCI host bridge to bus 0000:00 [ 0.298021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.301026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.304026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.306028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.309026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.311024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.313178] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.316027] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.320277] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.329019] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.337061] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.339021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.342021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.344020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.347491] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.348746] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.351047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.353827] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.359021] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.373921] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.378888] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.385012] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.393018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.400017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.420031] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.430060] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.438016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.445017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.465021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.475348] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.482014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.494021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.512026] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.522646] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.529014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.537017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.555020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.566025] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.573013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.579013] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.597019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.608937] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.621015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.628014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.644017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.655291] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.658406] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.660301] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.662353] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.665211] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.673043] iommu: Default domain type: Passthrough [ 0.674483] SCSI subsystem initialized [ 0.676169] ACPI: bus type USB registered [ 0.678088] usbcore: registered new interface driver usbfs [ 0.680082] usbcore: registered new interface driver hub [ 0.682090] usbcore: registered new device driver usb [ 0.684186] pps_core: LinuxPPS API ver. 1 registered [ 0.686013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.689058] PTP clock support registered [ 0.691158] EDAC MC: Ver: 3.0.0 [ 0.693122] PCI: Using ACPI for IRQ routing [ 0.694861] NetLabel: Initializing [ 0.696012] NetLabel: domain hash size = 128 [ 0.698008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.700079] NetLabel: unlabeled traffic allowed by default [ 0.702311] vgaarb: loaded [ 0.704276] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.706016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.713179] clocksource: Switched to clocksource kvm-clock [ 0.823316] VFS: Disk quotas dquot_6.6.0 [ 0.825099] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.828296] *** VALIDATE ramfs *** [ 0.829567] *** VALIDATE hugetlbfs *** [ 0.831311] pnp: PnP ACPI init [ 0.833854] pnp: PnP ACPI: found 6 devices [ 0.858816] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.862850] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.865181] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.867497] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.870237] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.872841] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.876058] NET: Registered protocol family 2 [ 0.878756] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.883696] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.887303] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.892663] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.896411] TCP: Hash tables configured (established 65536 bind 65536) [ 0.899739] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.903038] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.906142] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.909406] NET: Registered protocol family 1 [ 0.912528] RPC: Registered named UNIX socket transport module. [ 0.914957] RPC: Registered udp transport module. [ 0.916845] RPC: Registered tcp transport module. [ 0.918749] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.921371] NET: Registered protocol family 44 [ 0.923198] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.925555] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.927839] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.930334] PCI: CLS 0 bytes, default 64 [ 0.932174] Unpacking initramfs... [ 2.340976] debug: unmapping init [mem 0xffff8e9cfcc54000-0xffff8e9cfffbffff] [ 2.345107] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.347737] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.351043] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.864654] Initialise system trusted keyrings [ 2.866611] Key type blacklist registered [ 2.868628] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.877738] zbud: loaded [ 2.880892] *** VALIDATE nfs *** [ 2.882513] *** VALIDATE nfs4 *** [ 2.884261] pstore: using deflate compression [ 2.888748] Platform Keyring initialized [ 3.005683] NET: Registered protocol family 38 [ 3.008132] Key type asymmetric registered [ 3.009863] Asymmetric key parser 'x509' registered [ 3.011982] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.015529] io scheduler mq-deadline registered [ 3.017373] io scheduler kyber registered [ 3.019492] io scheduler bfq registered [ 3.021834] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.025220] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.028140] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.031411] ACPI: Power Button [PWRF] [ 3.041400] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.049477] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.066700] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.074489] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.089478] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.120453] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.158555] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.163596] Non-volatile memory driver v1.3 [ 3.165526] Linux agpgart interface v0.103 [ 3.200851] virtio_blk virtio1: [vda] 144824 512-byte logical blocks (74.1 MB/70.7 MiB) [ 3.203063] vda: detected capacity change from 0 to 74149888 [ 3.217140] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.219936] vdb: detected capacity change from 0 to 1073741824 [ 3.235704] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.238707] vdc: detected capacity change from 0 to 2621440000 [ 3.254156] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.257114] vdd: detected capacity change from 0 to 2621440000 [ 3.276069] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.279770] vde: detected capacity change from 0 to 4294967296 [ 3.299956] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.302792] vdf: detected capacity change from 0 to 4294967296 [ 3.312330] libphy: Fixed MDIO Bus: probed [ 3.321374] usbcore: registered new interface driver usbserial_generic [ 3.324390] usbserial: USB Serial support registered for generic [ 3.326940] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.332104] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.334117] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.336863] mousedev: PS/2 mouse device common for all mice [ 3.339724] rtc_cmos 00:05: RTC can wake from S4 [ 3.342367] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.344172] rtc_cmos 00:05: registered as rtc0 [ 3.348500] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.352399] intel_pstate: CPU model not supported [ 3.355371] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.359311] hid: raw HID events driver (C) Jiri Kosina [ 3.361821] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.361971] usbcore: registered new interface driver usbhid [ 3.367479] usbhid: USB HID core driver [ 3.369217] drop_monitor: Initializing network drop monitor service [ 3.371783] Initializing XFRM netlink socket [ 3.373874] NET: Registered protocol family 10 [ 3.377113] Segment Routing with IPv6 [ 3.378599] NET: Registered protocol family 17 [ 3.380287] mpls_gso: MPLS GSO support [ 3.386260] RAS: Correctable Errors collector initialized. [ 3.389143] AVX version of gcm_enc/dec engaged. [ 3.391121] AES CTR mode by8 optimization enabled [ 3.475650] sched_clock: Marking stable (3475616620, 0)->(4473711660, -998095040) [ 3.479907] registered taskstats version 1 [ 3.483154] Loading compiled-in X.509 certificates [ 3.485131] zswap: loaded using pool lzo/zbud [ 3.511990] Key type big_key registered [ 3.525593] Key type encrypted registered [ 3.527386] ima: No TPM chip found, activating TPM-bypass! [ 3.529659] ima: Allocated hash algorithm: sha1 [ 3.531534] ima: No architecture policies found [ 3.533519] evm: Initialising EVM extended attributes: [ 3.535608] evm: security.selinux [ 3.536953] evm: security.ima [ 3.538193] evm: security.capability [ 3.539676] evm: HMAC attrs: 0x1 [ 3.542057] rtc_cmos 00:05: setting system clock to 2026-06-28 01:04:52 UTC (1782608692) [ 3.548378] debug: unmapping init [mem 0xffffffff97403000-0xffffffff975fffff] [ 3.551367] debug: unmapping init [mem 0xffffffff96182000-0xffffffff96458fff] [ 3.558334] Write protecting the kernel read-only data: 28672k [ 3.562155] debug: unmapping init [mem 0xffffffff94803000-0xffffffff949fffff] [ 3.565537] debug: unmapping init [mem 0xffffffff95114000-0xffffffff951fffff] [ 3.602391] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.612593] systemd[1]: Detected virtualization kvm. [ 3.614844] systemd[1]: Detected architecture x86-64. [ 3.619141] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.645959] systemd[1]: No hostname configured. [ 3.647980] systemd[1]: Set hostname to . [ 3.650254] random: systemd: uninitialized urandom read (16 bytes read) [ 3.653738] systemd[1]: Initializing machine ID from random generator. [ 3.798667] random: systemd: uninitialized urandom read (16 bytes read) [ 3.801630] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.808122] random: systemd: uninitialized urandom read (16 bytes read) [ 3.810599] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.814784] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.528656] device-mapper: uevent: version 1.0.3 [ 4.530609] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.307448] random: fast init done [ 5.352497] virtio_net virtio0 ens2: renamed from eth0 [ 5.388161] scsi host0: ata_piix [ 5.451314] scsi host1: ata_piix [ 5.453252] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.455777] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.804849] dracut-initqueue[589]: RTNETLINK answers: File exists [ 10.044531] random: crng init done [ 10.046103] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.575807] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.879775] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.146142] SELinux: Disabled at runtime. [ 12.209201] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.218274] systemd[1]: Detected virtualization kvm. [ 12.220334] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.714135] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.717846] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.723151] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.728418] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.732321] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.743446] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.749401] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. [ OK [[ 12.798492] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS 0m] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Mounting Huge Pages File System... Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice system-getty.slice. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.301529] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.635147] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.728082] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.896323] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.915102] EDAC sbridge: Ver: 1.1.2 [ 15.785495] Key type dns_resolver registered [ 16.088573] NFS: Registering the id_resolver key type [ 16.090570] Key type id_resolver registered [ 16.092160] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg460-server login: [ 43.262226] libcfs: loading out-of-tree module taints kernel. [ 43.285967] Key type ._llcrypt registered [ 43.287881] Key type .llcrypt registered [ 43.339858] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 51.984153] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 52.842135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 52.852062] alg: No test for adler32 (adler32-zlib) [ 53.921198] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 54.289642] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [ 55.927195] Key type lgssc registered [ 56.712467] Lustre: Echo OBD driver; http://www.lustre.org/ [ 79.123340] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 82.243466] hrtimer: interrupt took 4662884 ns [ 129.456426] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 146.241669] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 146.332890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 147.850216] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 147.905946] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 148.121754] Lustre: lustre-MDT0000: new disk, initializing [ 148.279320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 148.313852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 153.383735] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 169.110745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 169.303636] Lustre: 6487:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 169.411642] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 169.418076] Lustre: Skipped 1 previous similar message [ 169.570278] Lustre: lustre-MDT0001: new disk, initializing [ 169.677982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 169.708232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 169.726159] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 176.055956] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 182.413377] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 195.928120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 196.404347] Lustre: lustre-OST0000: new disk, initializing [ 196.409783] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 196.425542] Lustre: 8427:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 196.534824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 203.808687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 203.821379] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 203.952307] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 203.991420] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 224.321641] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 224.508527] Lustre: lustre-OST0001: new disk, initializing [ 224.512288] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 224.528844] Lustre: 9500:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 224.672861] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 230.435538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 230.443529] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 230.515238] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 231.886980] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 246.235665] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 254.611528] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 262.923479] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing check_logdir /tmp/testlogs/ [ 268.649784] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing yml_node [ 273.497545] Lustre: DEBUG MARKER: Client: 2.17.54.105 [ 276.835776] Lustre: DEBUG MARKER: MDS: 2.17.54.105 [ 280.009717] Lustre: DEBUG MARKER: OSS: 2.17.54.105 [ 281.823196] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Jun 27 21:09:28 EDT 2026 [ 303.084215] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 305.032309] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 323.553466] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 323.561795] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 323.577435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 327.652472] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 327.657523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 327.664728] Lustre: Skipped 2 previous similar messages [ 327.692498] Lustre: Skipped 3 previous similar messages [ 332.770475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 332.778699] Lustre: Skipped 3 previous similar messages [ 337.375169] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 337.677195] Lustre: server umount lustre-MDT0000 complete [ 342.677471] LustreError: 6478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782609031 with bad export cookie 12754596815209484112 [ 342.685044] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 342.704169] LustreError: 6478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 343.010757] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 343.017138] LustreError: 6524:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 343.025373] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 343.068214] Lustre: Skipped 1 previous similar message [ 343.130578] LustreError: 6524:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 347.042536] LustreError: 6494:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 347.042991] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 347.088929] Lustre: Skipped 2 previous similar messages [ 349.535091] Lustre: server umount lustre-MDT0001 complete [ 355.078727] Lustre: server umount lustre-OST0000 complete [ 360.540573] Lustre: server umount lustre-OST0001 complete [ 369.269735] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 380.222479] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 424.994464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 425.668947] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 425.763207] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 425.991234] Lustre: lustre-MDT0000: new disk, initializing [ 426.127613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 426.164428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 432.143440] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 446.414228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 446.609267] Lustre: 15058:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 446.745491] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 446.754983] Lustre: Skipped 1 previous similar message [ 446.916164] Lustre: lustre-MDT0001: new disk, initializing [ 447.064536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 447.117933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 447.134440] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 453.279599] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 465.967200] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 473.605702] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 483.401683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 483.887498] Lustre: lustre-OST0000: new disk, initializing [ 483.892147] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 483.901589] Lustre: 17029:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 484.052371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 485.061358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 485.082747] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 485.231585] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 491.816835] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 503.607468] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 509.243518] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 509.551809] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 515.850142] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 516.173823] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 521.192569] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 521.195956] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 521.213051] LustreError: Skipped 1 previous similar message [ 521.220790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 521.238396] Lustre: Skipped 1 previous similar message [ 521.258769] Lustre: Skipped 1 previous similar message [ 524.938413] Lustre: server umount lustre-OST0000 complete [ 531.431166] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 531.454378] Lustre: Skipped 1 previous similar message [ 535.798738] Lustre: server umount lustre-MDT0000 complete [ 536.559746] LustreError: 15069:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 536.578441] LustreError: 15069:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 539.634915] LustreError: 15051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782609228 with bad export cookie 12754596815209485645 [ 539.641761] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 539.643768] LustreError: 15051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 540.037696] Lustre: server umount lustre-MDT0001 complete [ 546.349498] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 21:13:52 (1782609232) [ 567.802960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 568.421909] LustreError: 19904:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 568.506143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 573.920331] LustreError: 19905:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 574.153860] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 584.176491] LustreError: 19904:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 584.197954] LustreError: 19904:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 585.131906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 591.503746] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 600.642287] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 607.148692] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 609.733568] Lustre: Failing over lustre-MDT0000 [ 610.066980] Lustre: server umount lustre-MDT0000 complete [ 610.785566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 610.804820] Lustre: Skipped 1 previous similar message [ 610.805414] LustreError: 19899:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 610.834515] LustreError: 19899:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 626.148678] Lustre: 3620:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782609299/real 1782609299] req@ffff8e9c433e1f80 x1869200745474176/t0(0) o400->MGC192.168.204.160@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1782609315 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 626.182149] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 630.602377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 637.425811] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb1016e3a07e3dad3 [ 637.439532] Lustre: MGC192.168.204.160@tcp: Connection restored to 0@lo (at 0@lo) [ 637.691312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 637.698176] Lustre: Skipped 1 previous similar message [ 637.776846] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 643.051577] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 643.063644] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 643.124987] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 643.166362] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 657.843305] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 663.523958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 663.533417] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 663.547274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 663.550778] Lustre: Skipped 2 previous similar messages [ 669.845239] Lustre: server umount lustre-MDT0000 complete [ 672.754153] LustreError: 19899:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 672.789663] LustreError: 19899:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 674.578379] LustreError: 19884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782609363 with bad export cookie 12754596815209487059 [ 674.590711] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 674.600548] LustreError: 19884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 675.150192] Lustre: server umount lustre-MDT0001 complete [ 684.228505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 690.212523] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 700.890817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 707.364598] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 716.544105] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 723.193993] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 725.631539] Lustre: Failing over lustre-MDT0000 [ 725.665190] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 725.681664] Lustre: Skipped 1 previous similar message [ 725.691058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 725.696441] Lustre: Skipped 3 previous similar messages [ 725.986803] Lustre: server umount lustre-MDT0000 complete [ 737.249085] LustreError: 23245:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 737.280701] LustreError: 23245:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 742.303780] Lustre: 3619:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782609415/real 1782609415] req@ffff8e9d42f74380 x1869200745507456/t0(0) o400->MGC192.168.204.160@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1782609431 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 742.337859] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 747.149128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 752.609815] LustreError: 3618:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8e9c433e3480 x1869200745510016/t0(0) o250->MGC192.168.204.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 752.974680] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 758.109189] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 758.253477] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 758.258944] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 758.270896] Lustre: Skipped 1 previous similar message [ 758.292861] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 770.359985] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 781.985063] Lustre: server umount lustre-MDT0000 complete [ 787.033817] LustreError: 23228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782609475 with bad export cookie 12754596815209487997 [ 787.035319] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 787.043816] LustreError: 23228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 787.606396] Lustre: server umount lustre-MDT0001 complete [ 797.606817] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 21:18:04 (1782609484) [ 816.213963] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 826.940788] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 871.586573] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 886.189244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 886.502605] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 886.592976] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 886.706581] Lustre: lustre-MDT0000: new disk, initializing [ 886.827912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 886.841534] Lustre: Skipped 3 previous similar messages [ 886.855782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 893.230838] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 905.547130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 905.722288] Lustre: 29628:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 906.047473] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 910.935857] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 921.869189] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 928.625732] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 936.747417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 937.142617] Lustre: 31601:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 939.205571] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 939.246217] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 939.309938] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 939.327432] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 944.740348] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 955.463578] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 962.397439] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 990.696879] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 990.705816] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 990.710052] LustreError: Skipped 1 previous similar message [ 990.723636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 990.736049] Lustre: Skipped 4 previous similar messages [ 990.753418] Lustre: Skipped 3 previous similar messages [ 993.800474] Lustre: server umount lustre-OST0000 complete [ 998.368382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1004.492907] Lustre: server umount lustre-MDT0000 complete [ 1006.056120] LustreError: 30507:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1006.083728] LustreError: 30507:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 1008.484952] LustreError: 30433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782609697 with bad export cookie 12754596815209488396 [ 1008.494938] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1008.954481] Lustre: server umount lustre-MDT0001 complete [ 1022.041546] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [ 1025.860993] Key type lgssc unregistered [ 1026.203205] LNet: 33580:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1026.213169] LNetError: 33580:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1026.240521] LNet: Removed LNI 192.168.204.160@tcp [ 1027.307159] Key type .llcrypt unregistered [ 1027.316920] Key type ._llcrypt unregistered [ 1054.173459] Key type ._llcrypt registered [ 1054.177103] Key type .llcrypt registered [ 1054.311603] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 1055.564789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1055.640267] alg: No test for adler32 (adler32-zlib) [ 1056.808187] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 1057.064215] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [ 1058.720301] Key type lgssc registered [ 1059.922522] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1068.485876] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1068.524301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1070.026864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1075.257186] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 1079.285042] LustreError: 34996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1079.311896] LustreError: 34996:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1084.385322] LustreError: 34995:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1084.569671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1084.935183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1089.622071] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 1097.580627] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1103.932270] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1112.383949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1112.772603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1114.852510] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1119.683231] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 1128.189375] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1129.211137] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1142.145910] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 21:23:48 (1782609828) [ 1144.270661] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1146.309701] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1153.275809] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 21:23:59 (1782609839) [ 1162.534880] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 21:24:09 (1782609849) [ 1163.672268] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1173.337665] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1173.344711] Lustre: Skipped 2 previous similar messages [ 1174.947287] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 21:24:21 (1782609861) [ 1178.198724] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1178.206239] Lustre: Skipped 3 previous similar messages [ 1195.931262] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1195.939683] Lustre: Skipped 6 previous similar messages [ 1198.660821] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 21:24:45 (1782609885) [ 1201.445879] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1205.593231] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1216.398671] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 21:25:02 (1782609902) [ 1218.670486] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1218.673462] Lustre: Skipped 3 previous similar messages [ 1233.523660] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 21:25:19 (1782609919) [ 1252.325484] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1252.330861] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1252.334687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1252.344404] Lustre: Skipped 1 previous similar message [ 1257.441915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1257.452512] Lustre: Skipped 2 previous similar messages [ 1257.881225] Lustre: server umount lustre-OST0000 complete [ 1262.563389] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1262.584192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1262.592431] Lustre: Skipped 1 previous similar message [ 1267.680381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1267.685621] Lustre: Skipped 1 previous similar message [ 1268.454265] Lustre: server umount lustre-MDT0000 complete [ 1272.411672] LustreError: 34976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782609961 with bad export cookie 2055242868328351532 [ 1272.423081] LustreError: 34976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1272.427205] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1272.902414] Lustre: server umount lustre-MDT0001 complete [ 1285.746604] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [ 1288.754882] Key type lgssc unregistered [ 1289.104811] LNet: 42854:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1289.113129] LNetError: 42854:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1289.131798] LNet: Removed LNI 192.168.204.160@tcp [ 1289.960491] Key type .llcrypt unregistered [ 1289.962735] Key type ._llcrypt unregistered [ 1309.051286] Key type ._llcrypt registered [ 1309.054332] Key type .llcrypt registered [ 1309.172806] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 1327.111346] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 1328.199698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1328.545124] alg: No test for adler32 (adler32-zlib) [ 1329.742911] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 1330.005278] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [ 1331.695182] Key type lgssc registered [ 1332.975382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1371.223972] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1371.285816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1372.619978] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1372.686535] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1372.815380] Lustre: lustre-MDT0000: new disk, initializing [ 1372.916922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1372.939221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1377.502667] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 1389.155682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1389.263721] Lustre: 46008:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1389.303859] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1389.312466] Lustre: Skipped 1 previous similar message [ 1389.373663] Lustre: lustre-MDT0001: new disk, initializing [ 1389.454825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1389.510563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1389.529859] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1394.308325] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 1403.960679] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1410.141980] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1418.046378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1418.415405] Lustre: lustre-OST0000: new disk, initializing [ 1418.429412] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1418.438928] Lustre: 47980:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1418.596381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1425.436661] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 1426.558420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1426.567882] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1426.798972] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1436.565073] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1443.331558] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1443.655375] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1449.076330] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1449.311043] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1452.515196] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1452.520053] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1452.539977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1452.544710] Lustre: Skipped 1 previous similar message [ 1457.768786] Lustre: server umount lustre-OST0000 complete [ 1462.758509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1462.762840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1462.779793] Lustre: Skipped 1 previous similar message [ 1462.798746] Lustre: Skipped 2 previous similar messages [ 1467.881926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1467.886983] Lustre: Skipped 1 previous similar message [ 1468.797442] Lustre: server umount lustre-MDT0000 complete [ 1472.994068] LustreError: 47381:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1473.026094] LustreError: 47381:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1473.407586] LustreError: 46002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782610162 with bad export cookie 11179014014319762960 [ 1473.413564] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1473.420374] LustreError: 46002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1473.874216] Lustre: server umount lustre-MDT0001 complete [ 1484.344264] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 21:29:30 (1782610170) [ 1491.736160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1492.286378] LustreError: 50438:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1492.381572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1497.568641] LustreError: 50439:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1497.696078] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 1502.695164] LustreError: 50438:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1507.108633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1507.558073] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1512.532323] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 1520.956978] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1527.591626] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1535.348728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1535.727735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1543.431344] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 1551.933922] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1557.004777] LustreError: 52967:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 1558.128616] Lustre: Modifying parameter general.timeout=1 in log params [ 1560.112352] Lustre: Modifying parameter general.timeout=3 in log params [ 1560.132958] Lustre: Skipped 1 previous similar message [ 1562.192142] Lustre: Modifying parameter general.timeout=5 in log params [ 1562.195622] Lustre: Skipped 1 previous similar message [ 1566.386932] Lustre: Modifying parameter general.timeout=9 in log params [ 1566.398185] Lustre: Skipped 3 previous similar messages [ 1575.132536] Lustre: Modifying parameter general.timeout=17 in log params [ 1575.137368] Lustre: Skipped 7 previous similar messages [ 1591.287939] Lustre: Modifying parameter general.timeout=33 in log params [ 1591.294058] Lustre: Skipped 15 previous similar messages [ 1623.729613] Lustre: Modifying parameter general.timeout=63 in log params [ 1623.735330] Lustre: Skipped 29 previous similar messages [ 1688.060760] Lustre: Modifying parameter general.timeout=124 in log params [ 1688.071140] Lustre: Skipped 60 previous similar messages [ 1704.647604] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 21:33:11 (1782610391) [ 1707.726638] Lustre: 59919:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1709.717229] Lustre: 60018:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1711.914893] Lustre: 60133:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1716.186776] Lustre: 60331:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1716.198711] Lustre: 60331:0:(mgs_llog.c:1447:mgs_modify_param()) Skipped 1 previous similar message [ 1747.999494] Lustre: 61800:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1748.020129] Lustre: 61800:0:(mgs_llog.c:1447:mgs_modify_param()) Skipped 1 previous similar message [ 1748.054364] LustreError: 61800:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1753.276748] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 21:33:59 (1782610439) [ 1768.930153] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1768.944082] LustreError: Skipped 1 previous similar message [ 1768.953198] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1768.971182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1773.390946] Lustre: server umount lustre-MDT0000 complete [ 1774.565100] LustreError: 51167:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1774.581300] LustreError: 51167:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1777.835296] LustreError: 51173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782610466 with bad export cookie 11179014014319763842 [ 1777.850605] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1777.855842] LustreError: 51173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1778.336414] Lustre: server umount lustre-MDT0001 complete [ 1793.222671] Lustre: server umount lustre-OST0000 complete [ 1828.085803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1828.352206] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1828.395237] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1828.407158] Lustre: Skipped 28 previous similar messages [ 1828.747428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1833.870979] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 1842.966364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1843.125556] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1843.150149] Lustre: 64332:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1847.717299] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 1855.600854] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1862.703735] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1870.284370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1870.469947] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1870.826662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1870.839660] Lustre: Skipped 1 previous similar message [ 1877.932128] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 1886.604481] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1888.571269] Lustre: 66902:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1897.959052] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1897.964815] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1897.978474] Lustre: Skipped 2 previous similar messages [ 1897.982960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1897.985373] Lustre: Skipped 3 previous similar messages [ 1903.224077] Lustre: server umount lustre-OST0000 complete [ 1908.392360] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1908.397634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1908.417821] Lustre: Skipped 2 previous similar messages [ 1908.444298] Lustre: Skipped 3 previous similar messages [ 1915.149125] Lustre: server umount lustre-MDT0000 complete [ 1918.946083] LustreError: 65595:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1918.968363] LustreError: 65595:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1919.520102] LustreError: 64324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782610608 with bad export cookie 11179014014319767055 [ 1919.520899] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1919.538157] LustreError: 64324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1919.930284] Lustre: server umount lustre-MDT0001 complete [ 1931.859197] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [ 1935.476964] Key type lgssc unregistered [ 1936.030848] LNet: 68109:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1936.043991] LNetError: 68109:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1936.074185] LNet: Removed LNI 192.168.204.160@tcp [ 1937.116053] Key type .llcrypt unregistered [ 1937.120843] Key type ._llcrypt unregistered [ 1966.865095] Key type ._llcrypt registered [ 1966.867139] Key type .llcrypt registered [ 1967.023597] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 1968.308986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1968.485192] alg: No test for adler32 (adler32-zlib) [ 1969.804203] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 1970.128269] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [ 1971.871192] Key type lgssc registered [ 1973.221608] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1982.075049] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1982.120529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1983.913839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1989.218232] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 1993.187442] LustreError: 69525:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1993.215447] LustreError: 69525:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1998.305242] LustreError: 69526:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1999.125219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2000.256186] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2006.827097] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2017.760738] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2025.041663] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2033.799181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2034.864703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2044.710290] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2055.009338] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2065.308275] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2076.642486] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2076.659227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2076.692690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2079.201546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2079.218206] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2079.249663] Lustre: Skipped 1 previous similar message [ 2081.961065] Lustre: server umount lustre-MDT0000 complete [ 2084.323061] LustreError: 69526:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2084.358993] LustreError: 69526:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2086.828354] LustreError: 69507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782610775 with bad export cookie 3842296819201519243 [ 2086.832304] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2086.845070] LustreError: 69507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2087.281363] Lustre: server umount lustre-MDT0001 complete [ 2102.720195] Lustre: server umount lustre-OST0000 complete [ 2132.462194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2132.843978] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2132.881325] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2133.267711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2138.192989] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2147.429386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2147.559141] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2147.605935] Lustre: 74399:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2148.137231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2153.096158] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2161.385157] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2167.870873] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2175.760888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2176.113705] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2176.673763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2184.651978] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2193.790396] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2196.179738] Lustre: 76975:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2198.625356] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2206.702420] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2206.704158] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2206.704972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2206.704979] Lustre: Skipped 2 previous similar messages [ 2206.705738] LustreError: Skipped 1 previous similar message [ 2206.737737] Lustre: Skipped 1 previous similar message [ 2211.520135] Lustre: server umount lustre-OST0000 complete [ 2216.930740] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2216.945374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2216.954866] Lustre: Skipped 1 previous similar message [ 2216.961498] Lustre: Skipped 1 previous similar message [ 2222.049956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2222.059939] Lustre: Skipped 2 previous similar messages [ 2222.958110] Lustre: server umount lustre-MDT0000 complete [ 2226.147150] LustreError: 74411:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2226.173453] LustreError: 74411:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2227.467651] LustreError: 75145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782610916 with bad export cookie 3842296819201520090 [ 2227.476931] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2227.492512] LustreError: 75145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2227.791850] Lustre: server umount lustre-MDT0001 complete [ 2238.820246] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [ 2241.799866] Key type lgssc unregistered [ 2242.146335] LNet: 78193:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2242.159285] LNetError: 78193:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2242.177080] LNet: Removed LNI 192.168.204.160@tcp [ 2243.168159] Key type .llcrypt unregistered [ 2243.171226] Key type ._llcrypt unregistered [ 2270.951488] Key type ._llcrypt registered [ 2270.954137] Key type .llcrypt registered [ 2271.094700] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 2272.311260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2272.369646] alg: No test for adler32 (adler32-zlib) [ 2273.522917] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 2273.965542] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [ 2275.663191] Key type lgssc registered [ 2276.979261] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2286.404186] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2286.447461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2288.178657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2294.240059] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2298.342188] LustreError: 79612:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2298.386063] LustreError: 79612:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2303.467632] LustreError: 79611:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2304.922095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2305.866690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2311.214322] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2319.984922] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2327.737993] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2336.545880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2337.251501] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2344.388821] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2352.886249] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2363.582231] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2364.620603] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2365.893711] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2368.000680] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2368.005765] Lustre: Skipped 1 previous similar message [ 2372.653614] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2372.657466] Lustre: Skipped 4 previous similar messages [ 2381.081116] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2381.087203] Lustre: Skipped 8 previous similar messages [ 2398.010254] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2398.019959] Lustre: Skipped 15 previous similar messages [ 2430.770247] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2430.778752] Lustre: Skipped 32 previous similar messages [ 2472.301756] Lustre: 87720:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2475.822688] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 21:46:02 (1782611162) [ 2477.712819] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2480.001384] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 21:46:06 (1782611166) [ 2486.756594] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2486.787995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2486.795674] Lustre: Skipped 1 previous similar message [ 2486.817454] Lustre: Skipped 1 previous similar message [ 2491.893910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2491.922357] Lustre: Skipped 1 previous similar message [ 2497.001386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2497.009476] Lustre: Skipped 1 previous similar message [ 2499.039199] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2499.419327] Lustre: server umount lustre-OST0000 complete [ 2505.184166] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2505.189436] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2505.207994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2507.233923] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2510.105810] Lustre: server umount lustre-MDT0000 complete [ 2512.353620] LustreError: 80595:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2512.385019] LustreError: 80595:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2514.528910] LustreError: 79591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782611203 with bad export cookie 13785953069685484415 [ 2514.534108] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2514.549663] LustreError: 79591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2514.970629] Lustre: server umount lustre-MDT0001 complete [ 2528.201366] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [ 2531.359603] Key type lgssc unregistered [ 2531.784295] LNet: 88970:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2531.791433] LNetError: 88970:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2532.838138] LNet: Removed LNI 192.168.204.160@tcp [ 2533.898481] Key type .llcrypt unregistered [ 2533.911891] Key type ._llcrypt unregistered [ 2538.863688] Key type ._llcrypt registered [ 2538.867993] Key type .llcrypt registered [ 2538.960951] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2544.124219] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules [ 2545.483332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2545.607511] alg: No test for adler32 (adler32-zlib) [ 2546.824785] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 2547.117381] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [ 2548.863211] Key type lgssc registered [ 2550.649639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2552.603575] LustreError: 90212:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2570.942315] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 2573.608351] LustreError: 90212:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2580.107119] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2580.158514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2581.941582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2586.698498] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2591.200541] LustreError: 91203:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2591.235896] LustreError: 91203:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2596.313974] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 21:48:02 (1782611282) [ 2596.331503] LustreError: 91204:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2601.432058] LustreError: 91235:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2601.445204] LustreError: 91235:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2601.458946] LustreError: 91235:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 2602.129207] Lustre: server umount lustre-MDT0000 complete [ 2618.340583] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [ 2621.114907] Key type lgssc unregistered [ 2621.446918] LNet: 92730:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2621.457570] LNetError: 92730:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2622.503791] LNet: Removed LNI 192.168.204.160@tcp [ 2623.525544] Key type .llcrypt unregistered [ 2623.538356] Key type ._llcrypt unregistered [ 2645.243900] Key type ._llcrypt registered [ 2645.247408] Key type .llcrypt registered [ 2645.366080] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 2646.458071] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2646.666991] alg: No test for adler32 (adler32-zlib) [ 2647.778861] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 2648.037283] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [ 2649.783277] Key type lgssc registered [ 2651.269301] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2660.763836] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2660.807350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2662.318329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2667.599863] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2672.616847] LustreError: 93949:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2672.637028] LustreError: 93949:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2677.746034] LustreError: 93948:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2677.774627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2678.336419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2683.909316] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2692.012587] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2697.324938] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2704.803972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2705.566494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2712.659096] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2720.223490] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2778.755542] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 21:51:04 (1782611464) [ 2780.770666] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2782.751628] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 21:51:09 (1782611469) [ 2790.885772] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2790.895711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2790.905600] Lustre: Skipped 2 previous similar messages [ 2790.914146] Lustre: Skipped 1 previous similar message [ 2790.936213] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2793.113295] Lustre: server umount lustre-MDT0000 complete [ 2796.004364] LustreError: 93949:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2796.031457] LustreError: 93949:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2797.231138] LustreError: 93929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782611486 with bad export cookie 9850897228890298911 [ 2797.234369] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2797.249940] LustreError: 93929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2797.786935] Lustre: server umount lustre-MDT0001 complete [ 2811.588240] Lustre: server umount lustre-OST0000 complete [ 2828.740205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2829.549636] LustreError: 98246:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2829.954835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2835.426158] LustreError: 98247:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2836.392630] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2844.644977] LustreError: 98246:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2844.658741] LustreError: 98246:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2846.288423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2846.856072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2851.890868] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2860.590705] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2866.195813] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2880.011578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2880.384134] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2880.399952] LustreError: 98235:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2880.435528] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2880.444211] LustreError: 100329:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2880.452955] LustreError: 100329:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2880.458721] LustreError: 100329:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2880.548517] Lustre: server umount lustre-OST0000 complete [ 2880.551566] LustreError: 100329:0:(super25.c:185:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2890.265940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2890.399994] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2890.407897] LustreError: 98236:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2890.428491] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2890.436880] LustreError: 100798:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2890.443905] LustreError: 100798:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2890.449419] LustreError: 100798:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2890.548844] Lustre: server umount lustre-OST0000 complete [ 2890.551778] LustreError: 100798:0:(super25.c:185:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2899.798511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2900.156133] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2900.164952] Lustre: Found index 0 for lustre-OST0000, updating log [ 2900.183013] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2900.250620] Lustre: lustre-OST0000: new disk, initializing [ 2900.254961] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2900.261913] Lustre: 101334:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2900.557118] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2906.707844] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2910.424568] Lustre: Failing over lustre-OST0000 [ 2910.625438] Lustre: server umount lustre-OST0000 complete [ 2915.810123] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2915.822699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2915.833535] Lustre: Skipped 1 previous similar message [ 2916.900130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2916.911550] Lustre: Skipped 2 previous similar messages [ 2921.849102] Lustre: server umount lustre-MDT0000 complete [ 2921.957751] LustreError: 98247:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2921.981205] LustreError: 98247:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2926.189933] LustreError: 100350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782611615 with bad export cookie 9850897228890301655 [ 2926.192613] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2926.202562] LustreError: 100350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2926.661843] Lustre: server umount lustre-MDT0001 complete [ 2938.131957] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 21:53:44 (1782611624) [ 2959.867341] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 2970.305158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2970.680868] LustreError: 103680:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2970.806343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2976.208806] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2987.225705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2993.801857] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 2997.210300] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3005.182698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3006.459102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3006.475539] Lustre: Skipped 1 previous similar message [ 3015.200321] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 3026.600921] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3026.715810] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3026.796373] Lustre: lustre-OST0001: new disk, initializing [ 3026.800307] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3026.806262] Lustre: 106000:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3033.738295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3033.755585] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3033.877391] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3034.240599] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 3043.411090] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3052.699843] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3058.075869] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.60@tcp (stopping) [ 3059.702783] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3059.726688] Lustre: Skipped 3 previous similar messages [ 3063.133225] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.60@tcp (stopping) [ 3063.142648] Lustre: Skipped 3 previous similar messages [ 3063.880684] Lustre: server umount lustre-MDT0001 complete [ 3064.806626] LustreError: 103680:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3064.824196] LustreError: 103680:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3075.722154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3075.913276] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3075.925243] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3076.037230] Lustre: 103670:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3077.151389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3077.162343] Lustre: Skipped 1 previous similar message [ 3077.424738] LustreError: 107527:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3077.434192] LustreError: 107527:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3077.455221] LustreError: 103672:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3077.604700] LustreError: 107542:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.160@tcp: cfg command failed: rc = -17 [ 3077.622693] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3077.622693] [ 3082.243823] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3082.956673] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 3094.580753] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 21:56:20 (1782611780) [ 3118.325268] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 3138.382283] Lustre: 109634:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3162.633357] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3166.466699] Lustre: 110769:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3271.138582] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3271.148253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3271.192354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3274.215712] Lustre: server umount lustre-MDT0000 complete [ 3276.772582] LustreError: 103676:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3276.804711] LustreError: 103676:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 3279.134086] LustreError: 103660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782611968 with bad export cookie 9850897228890302642 [ 3279.138221] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3279.146326] LustreError: 103660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3279.673233] Lustre: server umount lustre-MDT0001 complete [ 3295.070448] Lustre: server umount lustre-OST0000 complete [ 3297.055218] Lustre: 93257:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782611970/real 1782611970] req@ffff8e9d6f7bb480 x1869203464636032/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782611986 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3297.097734] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3297.116099] Lustre: Skipped 3 previous similar messages [ 3299.808224] Lustre: server umount lustre-OST0001 complete [ 3307.649283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3323.781709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3335.497990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3352.499788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3380.970314] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 3394.122237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3394.165199] Lustre: lustre-MDT0000: reset Object Index mappings [ 3394.965801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3400.146172] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 3410.208707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3410.229275] Lustre: lustre-MDT0001: reset Object Index mappings [ 3417.273376] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 3421.402292] Lustre: 115885:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3429.524798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3435.758072] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:199 to 0x280000400:225) [ 3438.243743] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 3439.875544] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 3449.229671] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3454.979274] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3454.989416] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:157 to 0x2c0000401:193) [ 3457.798688] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 3467.363955] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3471.649918] Lustre: 117762:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3535.931661] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 22:03:42 (1782612222) [ 3564.512791] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3564.529121] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3564.550670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3564.553890] Lustre: Skipped 3 previous similar messages [ 3567.751580] Lustre: server umount lustre-MDT0000 complete [ 3570.670205] LustreError: 114738:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3570.696882] LustreError: 114738:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 3572.210859] LustreError: 117764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782612261 with bad export cookie 9850897228890349521 [ 3572.226046] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3578.886953] Lustre: server umount lustre-MDT0001 complete [ 3589.648775] Lustre: server umount lustre-OST0000 complete [ 3599.958541] Lustre: server umount lustre-OST0001 complete [ 3609.134249] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 3621.260184] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 3664.764869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3665.158648] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3665.192280] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3665.298479] Lustre: lustre-MDT0000: new disk, initializing [ 3665.361945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3665.369788] Lustre: Skipped 3 previous similar messages [ 3665.384714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3671.092515] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 3678.169166] Lustre: server umount lustre-MDT0000 complete [ 3690.003645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3690.626650] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3690.656965] Lustre: 122035:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3690.687042] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3695.990465] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 3701.762731] Lustre: server umount lustre-MDT0000 complete [ 3712.084360] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 22:06:38 (1782612398) [ 3713.978339] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3715.924730] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 22:06:42 (1782612402) [ 3742.501950] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 3755.752047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3761.868108] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 3771.802648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3771.911540] Lustre: 124108:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3771.963052] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3771.967236] Lustre: Skipped 1 previous similar message [ 3772.063165] Lustre: lustre-MDT0001: new disk, initializing [ 3772.238408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3772.255887] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3777.342519] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 3783.136638] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3790.785635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3791.271453] Lustre: lustre-OST0000: new disk, initializing [ 3791.277868] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3791.289524] Lustre: 125687:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3793.175119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3793.203624] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3793.357258] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3798.077596] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 3810.823701] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3811.039873] Lustre: lustre-OST0001: new disk, initializing [ 3811.046053] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3811.053162] Lustre: 126556:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3812.385338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3812.397921] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3812.549528] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3818.548354] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 3829.580573] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3834.646206] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4272.537827] Lustre: 132989:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4272.546073] Lustre: 132989:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4272.552764] Lustre: 132989:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4272.561731] Lustre: 132989:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4272.570349] Lustre: 132989:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4272.577792] Lustre: 132989:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4592.339907] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 22:21:18 (1782613278) [ 4611.555297] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4611.566169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4611.580898] Lustre: Skipped 8 previous similar messages [ 4611.599200] Lustre: Skipped 11 previous similar messages [ 4616.674259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4616.689080] Lustre: Skipped 3 previous similar messages [ 4622.303296] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4622.579789] Lustre: server umount lustre-MDT0000 complete [ 4626.501570] LustreError: 133016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782613315 with bad export cookie 9850897228890420683 [ 4626.507424] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4626.526412] LustreError: 133016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4626.914161] LustreError: 127133:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4626.918163] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4626.926345] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4626.926354] Lustre: Skipped 4 previous similar messages [ 4626.939618] LustreError: 127133:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4626.969228] Lustre: Skipped 1 previous similar message [ 4633.302511] Lustre: server umount lustre-MDT0001 complete [ 4651.999207] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4652.342702] Lustre: server umount lustre-OST0000 complete [ 4663.495218] Lustre: server umount lustre-OST0001 complete [ 4670.819271] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 4680.599907] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 4719.660869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4719.921163] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4719.977526] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4720.118697] Lustre: lustre-MDT0000: new disk, initializing [ 4720.242322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4720.260041] Lustre: Skipped 5 previous similar messages [ 4720.280511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4725.494951] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 4736.655571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4736.817670] Lustre: 136052:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4736.903134] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4736.908251] Lustre: Skipped 1 previous similar message [ 4736.983923] Lustre: lustre-MDT0001: new disk, initializing [ 4737.129206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4737.139669] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4742.107931] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 4752.135534] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4757.941208] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4765.566996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4765.921377] Lustre: lustre-OST0000: new disk, initializing [ 4765.928030] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4765.941756] Lustre: 138021:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4767.283348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4767.311757] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4767.477259] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4773.585649] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 4783.873588] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4794.732751] Lustre: lustre-MDD0000: changelog on [ 4797.443562] Lustre: lustre-MDD0001: changelog on [ 7127.571777] Lustre: lustre-MDD0001: changelog off [ 7130.029032] Lustre: 136929:0:(llog_cat.c:977:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 7131.350465] Lustre: lustre-MDD0000: changelog off [ 7141.359038] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7141.391545] Lustre: Skipped 1 previous similar message [ 7141.401410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7141.403892] Lustre: Skipped 3 previous similar messages [ 7145.679626] Lustre: server umount lustre-OST0000 complete [ 7151.585798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7151.619057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7156.704418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7156.714169] Lustre: Skipped 2 previous similar messages [ 7157.248944] Lustre: server umount lustre-MDT0000 complete [ 7161.831190] LustreError: 139039:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7161.860466] LustreError: 139039:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7162.330187] LustreError: 138020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782615851 with bad export cookie 9850897228891248426 [ 7162.349342] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7163.074293] Lustre: server umount lustre-MDT0001 complete [ 7176.528929] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [ 7179.509934] Key type lgssc unregistered [ 7179.882584] LNet: 147241:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7179.888941] LNetError: 147241:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7179.939346] LNet: Removed LNI 192.168.204.160@tcp [ 7180.855531] Key type .llcrypt unregistered [ 7180.861077] Key type ._llcrypt unregistered [ 7204.984833] Key type ._llcrypt registered [ 7204.987522] Key type .llcrypt registered [ 7205.141363] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 7221.909970] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 7223.040455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7223.105103] alg: No test for adler32 (adler32-zlib) [ 7224.176454] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 7224.501292] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [ 7226.191870] Key type lgssc registered [ 7227.493737] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7266.077938] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7266.127862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7267.422289] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7267.481405] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7267.591418] Lustre: lustre-MDT0000: new disk, initializing [ 7267.734951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7267.761821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7272.355322] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7282.971492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7283.109132] Lustre: 150446:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7283.165982] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7283.176852] Lustre: Skipped 1 previous similar message [ 7283.271480] Lustre: lustre-MDT0001: new disk, initializing [ 7283.421439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7283.479459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7283.491399] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7287.686402] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7297.038722] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7302.928222] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7310.498847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7310.911387] Lustre: lustre-OST0000: new disk, initializing [ 7310.918287] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7310.932182] Lustre: 152415:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7311.036802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7317.123635] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7320.156947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7320.175423] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7320.275112] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7326.614916] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7332.046931] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_module obdecho/obdecho [ 7334.169358] LustreError: 153418:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7334.179946] LustreError: 153418:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7335.970235] LustreError: 153514:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 7350.240907] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7350.250206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7350.263908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7351.271341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7351.272564] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7353.509767] Lustre: server umount lustre-MDT0000 complete [ 7356.931184] LustreError: 151252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782616045 with bad export cookie 1866830006666185488 [ 7356.948691] LustreError: 151252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7356.951624] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7357.306712] Lustre: server umount lustre-MDT0001 complete [ 7370.379773] Lustre: server umount lustre-OST0000 complete [ 7393.255481] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 7404.882213] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 7443.344450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7443.644132] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7443.690368] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7443.802078] Lustre: lustre-MDT0000: new disk, initializing [ 7443.902115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7443.918733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7448.050316] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7459.398276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7459.499472] Lustre: 157231:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7459.568073] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7459.580185] Lustre: Skipped 1 previous similar message [ 7459.660301] Lustre: lustre-MDT0001: new disk, initializing [ 7459.767409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7459.816039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7459.825935] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7464.270618] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7474.369466] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7480.287493] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7487.776336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7488.123918] Lustre: lustre-OST0000: new disk, initializing [ 7488.130210] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7488.150448] Lustre: 159198:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7488.246556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7495.735905] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7496.303103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7496.322573] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7496.511918] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7505.511431] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7510.248303] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7510.495752] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7515.569144] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7515.884930] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7522.272824] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7522.274883] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7522.289137] LustreError: Skipped 1 previous similar message [ 7522.293415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7522.304929] Lustre: Skipped 2 previous similar messages [ 7522.321133] Lustre: Skipped 3 previous similar messages [ 7524.087774] Lustre: server umount lustre-OST0000 complete [ 7531.488637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7531.501657] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7531.527031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7534.077951] Lustre: server umount lustre-MDT0000 complete [ 7537.634457] LustreError: 160508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7537.671111] LustreError: 160508:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7538.560868] LustreError: 157221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782616227 with bad export cookie 1866830006666186454 [ 7538.567331] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7538.575857] LustreError: 157221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7539.124326] Lustre: server umount lustre-MDT0001 complete [ 7543.816720] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 23:10:29 (1782616229) [ 7545.400942] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7547.470874] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 23:10:33 (1782616233) [ 7556.540271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7556.986472] LustreError: 161614:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7557.058578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7561.669470] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7562.212033] LustreError: 161615:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7562.246138] LustreError: 161615:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7566.323775] LustreError: 161614:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7570.515979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7575.467107] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7582.605843] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7588.437962] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7596.179246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7596.486367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7596.493966] Lustre: Skipped 1 previous similar message [ 7602.067356] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7609.282112] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7627.847376] Lustre: server umount lustre-MDT0000 complete [ 7630.728523] LustreError: 161609:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7631.332302] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7631.344696] Lustre: Skipped 3 previous similar messages [ 7632.289186] LustreError: 161596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782616321 with bad export cookie 1866830006666187336 [ 7632.296136] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7632.312335] LustreError: 161596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7634.668336] Lustre: server umount lustre-MDT0001 complete [ 7642.374338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7652.832502] Lustre: 148669:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782616325/real 1782616325] req@0000000019eb7812 x1869208263705216/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782616341 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7652.888904] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7653.856127] Lustre: 148670:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782616326/real 1782616326] req@000000007444a15a x1869208263705344/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782616342 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7656.351721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7656.599924] LustreError: 165476:0:(llog.c:1646:llog_backup()) MGC192.168.204.160@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7656.617582] Lustre: 165476:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.160@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7657.951144] Lustre: 148672:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782616330/real 1782616330] req@00000000c65739ef x1869208263705472/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782616346 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7661.466565] LustreError: 165481:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7661.484122] LustreError: 165481:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7663.085292] Lustre: 148669:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782616336/real 1782616336] req@0000000093f6c765 x1869208263705856/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782616352 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7663.122900] Lustre: 148669:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7667.167514] Lustre: 148670:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782616340/real 1782616340] req@00000000c389cc57 x1869208263706112/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782616356 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7681.892905] LustreError: 165482:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7681.905035] LustreError: 165482:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 16 previous similar messages [ 7716.696404] LustreError: 165483:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7716.727847] LustreError: 165483:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 24 previous similar messages [ 7717.856424] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7717.865733] Lustre: 165476:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.204.160@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7758.846073] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7763.638381] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7763.942912] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7772.203615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7777.265408] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x19e850000b60eb7e [ 7777.284616] Lustre: MGC192.168.204.160@tcp: Connection restored to 0@lo (at 0@lo) [ 7781.460898] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7782.904214] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7788.410289] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7790.201159] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7795.209358] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7796.719556] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7818.721336] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7818.722135] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7818.729195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7818.735485] Lustre: Skipped 2 previous similar messages [ 7818.748888] Lustre: Skipped 4 previous similar messages [ 7818.953963] Lustre: server umount lustre-MDT0000 complete [ 7822.383381] LustreError: 161596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782616511 with bad export cookie 1866830006666193790 [ 7822.390041] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7822.400294] LustreError: 161596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7822.758801] Lustre: server umount lustre-MDT0001 complete [ 7837.514239] Lustre: server umount lustre-OST0000 complete [ 7860.022183] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 7873.350397] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 7916.131155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7916.409342] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7916.477337] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7916.581803] Lustre: lustre-MDT0000: new disk, initializing [ 7916.663737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7916.669126] Lustre: Skipped 1 previous similar message [ 7916.698518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7921.255710] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7934.425750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7934.635438] Lustre: 170586:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7934.971231] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7941.038344] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7953.783713] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7962.824400] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7973.351899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7973.817633] Lustre: lustre-OST0000: new disk, initializing [ 7973.826313] Lustre: Skipped 1 previous similar message [ 7973.833396] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7973.844429] Lustre: Skipped 2 previous similar messages [ 7973.853451] Lustre: 172554:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7980.637567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7980.654046] Lustre: Skipped 1 previous similar message [ 7980.663203] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7980.802138] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7982.735874] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 7993.203654] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7999.217237] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7999.586555] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8004.693913] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8005.015947] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8011.744865] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8011.751449] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8011.756797] LustreError: Skipped 1 previous similar message [ 8011.759666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8011.778281] Lustre: Skipped 1 previous similar message [ 8011.803893] Lustre: Skipped 1 previous similar message [ 8013.636920] Lustre: server umount lustre-OST0000 complete [ 8021.992935] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8023.995486] Lustre: server umount lustre-MDT0000 complete [ 8027.105759] LustreError: 170591:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8027.126173] LustreError: 170591:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 46 previous similar messages [ 8028.394026] LustreError: 170577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782616717 with bad export cookie 1866830006666210618 [ 8028.404271] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8028.418545] LustreError: 170577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8029.152765] Lustre: server umount lustre-MDT0001 complete [ 8039.177231] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 23:18:45 (1782616725) [ 8045.861781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8050.919993] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8060.135871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8064.699725] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8072.710213] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8078.452751] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8084.872822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8091.735586] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8099.815473] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8111.580189] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 23:19:58 (1782616798) [ 8117.218092] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8117.227365] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8117.241945] LustreError: Skipped 1 previous similar message [ 8117.242885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8117.242890] Lustre: Skipped 2 previous similar messages [ 8117.288987] Lustre: Skipped 3 previous similar messages [ 8122.641587] Lustre: server umount lustre-OST0000 complete [ 8133.421707] Lustre: server umount lustre-MDT0000 complete [ 8137.342846] LustreError: 174949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782616826 with bad export cookie 1866830006666211500 [ 8137.344879] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8137.354956] LustreError: 174949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8137.597481] Lustre: server umount lustre-MDT0001 complete [ 8150.480865] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [ 8153.874758] Key type lgssc unregistered [ 8154.224685] LNet: 178698:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8154.243193] LNetError: 178698:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8154.275679] LNet: Removed LNI 192.168.204.160@tcp [ 8155.451112] Key type .llcrypt unregistered [ 8155.453734] Key type ._llcrypt unregistered [ 8177.292283] Key type ._llcrypt registered [ 8177.295883] Key type .llcrypt registered [ 8177.433308] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 8178.625119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8178.704715] alg: No test for adler32 (adler32-zlib) [ 8179.857407] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 8180.196023] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [ 8181.906794] Key type lgssc registered [ 8183.009709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8190.945306] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8190.984464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8206.565176] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8211.679342] LustreError: 179944:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.160@tcp: failed processing log, type 4: rc = -110 [ 8216.863585] LustreError: 179944:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8216.881172] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8216.893987] LustreError: MGC192.168.204.160@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8216.913203] LustreError: 179944:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8216.928466] LustreError: 179944:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 8216.934895] LustreError: 179944:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8217.012166] Lustre: server umount lustre-OST0000 complete [ 8217.016157] LustreError: 179944:0:(super25.c:185:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8226.169396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8227.620368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8232.562773] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8232.928127] LustreError: 180444:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8232.950457] LustreError: 180444:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8238.048423] LustreError: 180445:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8242.616092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8243.212778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8247.987347] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8255.869510] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8262.181708] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8269.573768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8269.764632] LustreError: 182261:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8269.786082] Lustre: 182261:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.160@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8269.958151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8277.052961] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8285.594988] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8288.738365] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8288.742335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8288.765458] Lustre: Skipped 2 previous similar messages [ 8288.787029] Lustre: Skipped 2 previous similar messages [ 8289.834847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8289.844079] Lustre: Skipped 2 previous similar messages [ 8294.640707] Lustre: server umount lustre-MDT0000 complete [ 8294.880405] LustreError: 180445:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8294.919908] LustreError: 180445:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8298.792232] LustreError: 180424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782616987 with bad export cookie 15621063515368917651 [ 8298.802553] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8298.814287] LustreError: 180424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8300.000728] LustreError: 180439:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8300.004992] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8300.041995] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8300.191321] LustreError: 181173:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000d636192a x1869209266057728/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8300.456065] Lustre: server umount lustre-MDT0001 complete [ 8307.423554] Lustre: server umount lustre-OST0000 complete [ 8317.446039] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 8330.054560] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 8374.256048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8374.640393] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8374.676236] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8374.824048] Lustre: lustre-MDT0000: new disk, initializing [ 8374.996557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8375.027474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8380.335194] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8391.568338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8391.701457] Lustre: 185783:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8391.803302] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8391.810173] Lustre: Skipped 1 previous similar message [ 8391.902078] Lustre: lustre-MDT0001: new disk, initializing [ 8392.000650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8392.053017] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8392.070864] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8396.924793] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8407.578231] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8415.110971] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8424.551673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8424.796945] Lustre: lustre-OST0000: new disk, initializing [ 8424.801393] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8424.810865] Lustre: 187753:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8424.926545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8432.968308] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8433.767825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8433.786436] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8433.967695] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8443.892198] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8449.373923] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8449.584388] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8454.191554] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8454.486086] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8459.747670] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8459.753123] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8459.764359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8459.779131] Lustre: Skipped 1 previous similar message [ 8459.790184] Lustre: Skipped 1 previous similar message [ 8464.872400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8464.878856] Lustre: Skipped 1 previous similar message [ 8471.007167] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8471.162406] Lustre: server umount lustre-OST0000 complete [ 8479.203574] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8479.221697] LustreError: Skipped 1 previous similar message [ 8479.229271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8479.246116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8479.253394] Lustre: Skipped 2 previous similar messages [ 8481.939840] Lustre: server umount lustre-MDT0000 complete [ 8485.344530] LustreError: 185820:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8485.360353] LustreError: 185820:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8486.319304] LustreError: 185777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782617175 with bad export cookie 15621063515368918372 [ 8486.335106] LustreError: 185777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8486.336282] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8486.903635] Lustre: server umount lustre-MDT0001 complete [ 8496.080344] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 23:26:22 (1782617182) [ 8515.425351] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [ 8518.628764] Key type lgssc unregistered [ 8518.950398] LNet: 190703:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8518.969793] LNetError: 190703:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8519.005116] LNet: Removed LNI 192.168.204.160@tcp [ 8520.018079] Key type .llcrypt unregistered [ 8520.021934] Key type ._llcrypt unregistered [ 8540.173273] Key type ._llcrypt registered [ 8540.177200] Key type .llcrypt registered [ 8540.315647] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 8541.480434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8541.552549] alg: No test for adler32 (adler32-zlib) [ 8542.673410] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 8542.929768] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [ 8544.600741] Key type lgssc registered [ 8545.763827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8554.146638] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8554.164680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8555.702265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8560.262875] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8565.216840] LustreError: 191927:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8565.235295] LustreError: 191927:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8569.331267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8569.590747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8574.480946] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8582.466699] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8588.661594] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8598.548344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8598.736093] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8598.741755] LustreError: 193792:0:(llog.c:1687:llog_backup()) MGC192.168.204.160@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8598.751220] Lustre: 193792:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.160@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8598.915505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8607.820893] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8617.457164] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8620.527210] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8620.529780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8620.542774] Lustre: Skipped 1 previous similar message [ 8620.567961] Lustre: Skipped 2 previous similar messages [ 8625.653268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8625.659891] Lustre: Skipped 2 previous similar messages [ 8626.718351] Lustre: server umount lustre-MDT0000 complete [ 8630.754817] LustreError: 191927:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8630.792523] LustreError: 191927:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8631.284711] LustreError: 191907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782617320 with bad export cookie 5208522409985368670 [ 8631.296131] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8631.841711] Lustre: server umount lustre-MDT0001 complete [ 8647.099970] Lustre: server umount lustre-OST0000 complete [ 8656.730364] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 8668.252473] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 8711.732790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8712.155479] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8712.210398] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8712.291820] Lustre: lustre-MDT0000: new disk, initializing [ 8712.393833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8712.421053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8717.218495] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8728.843930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8728.974527] Lustre: 197321:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8729.022837] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8729.024789] Lustre: Skipped 1 previous similar message [ 8729.033662] LustreError: 198104:0:(llog.c:1687:llog_backup()) MGC192.168.204.160@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8729.040403] LustreError: 198104:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 8729.052074] Lustre: 198104:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.160@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8729.075601] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8729.079101] Lustre: Skipped 1 previous similar message [ 8729.186584] Lustre: lustre-MDT0001: new disk, initializing [ 8729.310602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8729.370254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8729.386794] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8734.219517] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8744.756690] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8752.097858] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8759.816776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8760.178621] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8760.181603] Lustre: Skipped 1 previous similar message [ 8760.186776] LustreError: 199267:0:(llog.c:1687:llog_backup()) MGC192.168.204.160@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8760.198389] LustreError: 199267:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 8760.204482] Lustre: 199267:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.160@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8760.213296] Lustre: 199267:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8760.242582] Lustre: lustre-OST0000: new disk, initializing [ 8760.246574] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8760.253146] Lustre: 199290:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8760.348592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8766.534468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8766.551341] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8766.689777] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8767.613843] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8778.526960] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8784.247614] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8784.478365] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8789.772633] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8790.062919] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8792.551545] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8792.563035] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8792.574850] Lustre: Skipped 2 previous similar messages [ 8792.578574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8792.582267] LustreError: Skipped 1 previous similar message [ 8792.608828] Lustre: Skipped 1 previous similar message [ 8797.667833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8797.674871] Lustre: Skipped 1 previous similar message [ 8798.553794] Lustre: server umount lustre-OST0000 complete [ 8802.784174] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8802.788134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8802.808815] Lustre: Skipped 1 previous similar message [ 8802.823698] Lustre: Skipped 1 previous similar message [ 8809.005560] Lustre: server umount lustre-MDT0000 complete [ 8813.029161] LustreError: 198131:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8813.053147] LustreError: 198131:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8813.595705] LustreError: 197314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782617502 with bad export cookie 5208522409985369391 [ 8813.606303] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8813.608799] LustreError: 197314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8813.919959] Lustre: server umount lustre-MDT0001 complete [ 8823.944103] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 23:31:50 (1782617510) [ 8842.282181] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 8853.061358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8853.468314] LustreError: 202420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8853.545724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8858.091903] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8858.595794] LustreError: 202421:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8866.647130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8871.113856] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8873.984938] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8881.543526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8888.143941] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8897.583856] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8897.807459] Lustre: lustre-OST0001: new disk, initializing [ 8897.811781] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8897.821254] Lustre: 204727:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8897.940106] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8897.946153] Lustre: Skipped 2 previous similar messages [ 8905.713645] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8907.462655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8907.520100] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8907.737681] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8918.275971] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8927.419070] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8929.705349] Lustre: DEBUG MARKER: ADD OST3 [ 8935.102684] loop: module loaded [ 8935.171673] loop0: detected capacity change from 0 to 204800000 [ 8935.500632] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8935.532910] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8935.553421] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8942.368858] loop0: detected capacity change from 0 to 204800000 [ 8943.253800] Lustre: DEBUG MARKER: STOP OST3 [ 8950.846981] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8951.167779] Lustre: lustre-OST0003: new disk, initializing [ 8951.178713] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8951.207441] Lustre: 206760:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8952.724691] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8952.938097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8952.963253] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8953.005805] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8954.783599] LustreError: 206886:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8955.272314] Lustre: Failing over lustre-OST0003 [ 8955.600644] Lustre: server umount lustre-OST0003 complete [ 8957.816629] LustreError: 203906:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8957.854382] LustreError: 203906:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8957.944411] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8957.949707] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8958.000456] Lustre: Skipped 1 previous similar message [ 8960.339450] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 8971.906452] Lustre: DEBUG MARKER: START OST3 again [ 8978.306714] LustreError: 205258:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8978.332948] LustreError: 205258:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 8980.545456] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8981.079236] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8981.085139] Lustre: Skipped 1 previous similar message [ 8981.095896] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8982.442985] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8982.682310] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8982.689169] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8988.647770] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9011.008795] Lustre: Failing over lustre-OST0003 [ 9011.227754] Lustre: server umount lustre-OST0003 complete [ 9011.689800] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9011.703412] LustreError: 205151:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9011.712418] Lustre: Skipped 1 previous similar message [ 9011.751771] LustreError: 205151:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9030.631513] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9030.646714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9030.679403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9030.689173] Lustre: Skipped 2 previous similar messages [ 9036.982510] Lustre: server umount lustre-MDT0000 complete [ 9043.534132] LustreError: 206756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782617732 with bad export cookie 5208522409985370273 [ 9043.568077] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9043.580951] LustreError: 206756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9047.541828] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9047.556634] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9047.566300] Lustre: Skipped 4 previous similar messages [ 9047.579464] Lustre: Skipped 4 previous similar messages [ 9050.224654] Lustre: server umount lustre-MDT0001 complete [ 9055.005984] Lustre: server umount lustre-OST0000 complete [ 9062.065306] Lustre: server umount lustre-OST0001 complete [ 9070.240289] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 9082.073829] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 9133.111954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9133.383554] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9133.420972] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9133.548832] Lustre: lustre-MDT0000: new disk, initializing [ 9133.672748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9133.696984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9139.334085] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9150.775984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9150.886838] Lustre: 212189:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9150.942443] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9150.948196] Lustre: Skipped 1 previous similar message [ 9151.035603] Lustre: lustre-MDT0001: new disk, initializing [ 9151.185351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9151.199889] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9156.490445] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9168.243320] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9175.595966] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9183.765782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9184.204491] Lustre: lustre-OST0000: new disk, initializing [ 9184.209347] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9184.221154] Lustre: 214161:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9186.292259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9186.307422] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9186.518373] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9192.095655] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9202.409110] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9207.936430] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9208.164839] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9213.842877] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9214.214857] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9216.992574] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9217.004126] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9217.008891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9217.008899] Lustre: Skipped 3 previous similar messages [ 9217.059620] Lustre: Skipped 1 previous similar message [ 9222.854259] Lustre: server umount lustre-OST0000 complete [ 9228.260491] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9242.079141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9242.314865] Lustre: server umount lustre-MDT0000 complete [ 9242.605829] LustreError: 212202:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9242.631058] LustreError: 212202:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 22 previous similar messages [ 9246.763185] LustreError: 212183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782617935 with bad export cookie 5208522409985372569 [ 9246.777579] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9246.790925] LustreError: 212183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9247.256493] Lustre: server umount lustre-MDT0001 complete [ 9252.253449] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 23:38:57 (1782617937) [ 9272.768754] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 9285.629314] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 9327.082821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9327.328961] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9327.368742] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9327.459741] Lustre: lustre-MDT0000: new disk, initializing [ 9327.555797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9333.319361] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9344.343607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9344.563175] Lustre: 218775:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9344.856286] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9349.782161] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9359.780721] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9367.323413] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9374.417607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9374.721937] Lustre: 220745:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9376.047412] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9381.327352] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9392.247789] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9397.677927] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9398.160383] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9406.066981] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9406.445922] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9412.064971] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9412.066425] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9412.089815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9412.089822] Lustre: Skipped 8 previous similar messages [ 9412.148808] Lustre: Skipped 3 previous similar messages [ 9415.657267] Lustre: server umount lustre-OST0000 complete [ 9427.193385] Lustre: server umount lustre-MDT0000 complete [ 9427.424902] LustreError: 219653:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9427.446969] LustreError: 219653:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9431.587516] LustreError: 218769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782618120 with bad export cookie 5208522409985373451 [ 9431.616695] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9431.941323] Lustre: server umount lustre-MDT0001 complete [ 9440.192213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9440.819202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9440.828342] Lustre: Skipped 5 previous similar messages [ 9445.504175] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9454.242853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9459.098947] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9467.531801] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9473.688345] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9480.662646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9487.239704] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9495.226325] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9505.771873] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9505.784542] LustreError: Skipped 1 previous similar message [ 9511.328746] Lustre: server umount lustre-OST0000 complete [ 9530.335220] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9530.609992] Lustre: server umount lustre-MDT0000 complete [ 9534.412565] LustreError: 222943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782618223 with bad export cookie 5208522409985374333 [ 9534.420547] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9534.421468] LustreError: 222943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9534.772380] Lustre: server umount lustre-MDT0001 complete [ 9637.855588] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [ 9640.760667] Key type lgssc unregistered [ 9641.043570] LNet: 226628:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9641.053764] LNetError: 226628:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9641.074420] LNet: Removed LNI 192.168.204.160@tcp [ 9642.359802] Key type .llcrypt unregistered [ 9642.367483] Key type ._llcrypt unregistered [ 9670.851774] Key type ._llcrypt registered [ 9670.856640] Key type .llcrypt registered [ 9670.974909] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [ 9686.843532] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 9688.391256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9688.444586] alg: No test for adler32 (adler32-zlib) [ 9689.561872] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 9689.883049] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [ 9691.559179] Key type lgssc registered [ 9692.836161] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9732.004236] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9732.030144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9733.369213] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9733.421303] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9733.482247] Lustre: lustre-MDT0000: new disk, initializing [ 9733.621994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9733.666114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9738.814513] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9751.379218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9751.579135] Lustre: 229982:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9751.692765] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9751.698207] Lustre: Skipped 1 previous similar message [ 9751.813756] Lustre: lustre-MDT0001: new disk, initializing [ 9751.897087] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9751.934770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9751.952083] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9758.412695] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9769.561726] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9776.202643] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9784.621325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9784.938916] Lustre: lustre-OST0000: new disk, initializing [ 9784.942691] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9784.960345] Lustre: 231952:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9785.062623] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9791.089056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9791.108978] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9791.304985] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9793.049636] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9804.863312] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9810.614515] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9810.812905] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9815.244364] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9815.471728] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9822.183568] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9822.184609] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9822.190132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9822.196524] LustreError: Skipped 1 previous similar message [ 9822.233394] Lustre: Skipped 1 previous similar message [ 9823.963491] Lustre: server umount lustre-OST0000 complete [ 9828.833099] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9828.842904] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9828.856815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9828.861078] Lustre: Skipped 1 previous similar message [ 9832.418244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9832.422821] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9834.256401] Lustre: server umount lustre-MDT0000 complete [ 9838.424935] LustreError: 229976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782618527 with bad export cookie 5385334250394468277 [ 9838.431436] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9838.436194] LustreError: 229976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9838.932477] Lustre: server umount lustre-MDT0001 complete [ 9861.308632] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [ 9873.585890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9874.093196] LustreError: 234888:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9874.113975] LustreError: 234888:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9874.138257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9878.777134] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9879.523252] LustreError: 234889:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9884.646135] LustreError: 234888:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9888.455063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9888.925672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9893.758850] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9896.893137] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9903.762953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9904.233866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9911.141663] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9920.098604] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9920.248289] Lustre: lustre-OST0001: new disk, initializing [ 9920.251950] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9920.257234] Lustre: 237197:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9920.344225] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9926.651652] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [ 9928.275435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9928.285140] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9928.374042] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9937.773983] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9944.092951] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9958.559583] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 23:50:45 (1782618645) [ 9964.513894] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9964.521429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9964.527174] Lustre: Skipped 2 previous similar messages [ 9964.537500] Lustre: Skipped 4 previous similar messages [ 9969.638070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9969.658318] Lustre: Skipped 3 previous similar messages [ 9975.263195] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9975.504173] Lustre: server umount lustre-MDT0000 complete [ 9978.922559] LustreError: 237963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782618667 with bad export cookie 5385334250394469159 [ 9978.926227] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9978.939347] LustreError: 237963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9979.255507] Lustre: server umount lustre-MDT0001 complete [ 9993.642801] Lustre: server umount lustre-OST0000 complete [ 9995.551195] Lustre: 228205:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782618668/real 1782618668] req@ffff8e9d7795e680 x1869210848447232/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782618684 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9995.589845] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9995.606379] Lustre: Skipped 1 previous similar message [ 9997.420895] Lustre: server umount lustre-OST0001 complete [10003.977380] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [10015.984846] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [10056.040415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10056.241200] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10056.262133] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10056.387436] Lustre: lustre-MDT0000: new disk, initializing [10056.560476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10056.600401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10060.884950] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [10071.300979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10071.424304] Lustre: 241140:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10071.475395] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10071.479330] Lustre: Skipped 1 previous similar message [10071.537164] Lustre: lustre-MDT0001: new disk, initializing [10071.669910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10071.679987] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10075.959125] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [10085.603547] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10091.021390] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10097.911369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10098.373489] Lustre: lustre-OST0000: new disk, initializing [10098.378883] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10098.390765] Lustre: 243110:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10099.787646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10099.796018] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10099.897940] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10105.682438] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [10115.806505] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10120.794874] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10121.011612] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10125.553741] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10125.926263] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10130.917069] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10130.918579] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10130.926192] LustreError: Skipped 1 previous similar message [10130.933105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10130.937391] Lustre: Skipped 1 previous similar message [10130.943160] Lustre: Skipped 4 previous similar messages [10142.176746] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10142.549121] Lustre: server umount lustre-OST0000 complete [10148.328780] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10148.342987] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10148.379119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10148.383333] Lustre: Skipped 5 previous similar messages [10153.950364] Lustre: server umount lustre-MDT0000 complete [10156.514749] LustreError: 241152:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10156.558404] LustreError: 241152:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10158.123606] LustreError: 241135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782618847 with bad export cookie 5385334250394471294 [10158.126416] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10158.138176] LustreError: 241135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10158.480407] Lustre: server umount lustre-MDT0001 complete [10166.557094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10166.998296] LustreError: 245330:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10167.078896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10167.082114] Lustre: Skipped 2 previous similar messages [10171.765380] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [10176.488913] LustreError: 245330:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10176.504561] LustreError: 245330:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10179.990938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10184.129185] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [10191.369746] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10197.818332] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10204.887785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10205.089839] LustreError: 247150:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10211.081413] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [10218.249263] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10364.898105] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10364.910920] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10364.918620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10364.918628] Lustre: Skipped 2 previous similar messages [10364.951433] Lustre: Skipped 2 previous similar messages [10370.877228] Lustre: server umount lustre-OST0000 complete [10375.142822] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10381.258109] Lustre: server umount lustre-MDT0000 complete [10385.377204] LustreError: 247769:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10385.407361] LustreError: 247769:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10385.516569] LustreError: 245311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782619074 with bad export cookie 5385334250394472176 [10385.533912] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10385.909183] Lustre: server umount lustre-MDT0001 complete [10491.454442] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [10495.511371] Key type lgssc unregistered [10495.859649] LNet: 248939:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10495.872452] LNetError: 248939:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10495.894385] LNet: Removed LNI 192.168.204.160@tcp [10497.220151] Key type .llcrypt unregistered [10497.226753] Key type ._llcrypt unregistered [10528.877499] Key type ._llcrypt registered [10528.879499] Key type .llcrypt registered [10529.035191] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [10546.162996] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [10547.598354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10547.615873] alg: No test for adler32 (adler32-zlib) [10548.749718] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [10549.036654] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [10550.743168] Key type lgssc registered [10552.217811] Lustre: Echo OBD driver; http://www.lustre.org/ [10607.384039] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [10618.873465] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10618.891430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10620.179973] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10620.274802] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10620.388522] Lustre: lustre-MDT0000: new disk, initializing [10620.525667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10620.585458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10626.124500] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [10640.211459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10640.409727] Lustre: 253028:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10640.506499] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10640.524466] Lustre: Skipped 1 previous similar message [10640.684232] Lustre: lustre-MDT0001: new disk, initializing [10640.910457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10640.989638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10641.008968] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10645.705469] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [10650.904727] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10657.886055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10658.265530] Lustre: lustre-OST0000: new disk, initializing [10658.271929] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10658.278979] Lustre: 254659:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10658.398238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10659.291496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10659.306659] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10659.462524] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10665.441084] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [10676.556892] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10676.740639] Lustre: lustre-OST0001: new disk, initializing [10676.744698] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10676.752516] Lustre: 255531:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10676.863899] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10683.447637] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [10685.539948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10685.550861] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10685.628874] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10694.598487] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10704.707244] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10717.674414] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10717.688949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10717.705083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10721.768176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10721.786378] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10723.216928] Lustre: server umount lustre-MDT0000 complete [10727.022269] LustreError: 254658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782619415 with bad export cookie 15242988958324499711 [10727.029867] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10727.033404] LustreError: 254658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10727.323621] Lustre: server umount lustre-MDT0001 complete [10741.358823] Lustre: server umount lustre-OST0000 complete [10755.769475] Lustre: server umount lustre-OST0001 complete [10770.030340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10779.127394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10787.401997] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10796.729136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10817.001984] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [10827.615805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10827.646655] Lustre: lustre-MDT0000: reset Object Index mappings [10828.216601] LustreError: 260544:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10828.236200] LustreError: 260544:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10828.339328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10833.331265] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [10833.380201] LustreError: 260545:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10838.508551] LustreError: 260544:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10843.361215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10843.376842] Lustre: lustre-MDT0001: reset Object Index mappings [10843.617300] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10843.631375] Lustre: Skipped 3 previous similar messages [10843.762416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10848.981703] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [10852.411439] Lustre: 261686:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10860.000653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10860.527198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10867.478634] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [10868.719973] LustreError: 262042:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10873.840575] LustreError: 262040:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10873.861501] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10876.322612] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10882.071067] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10884.227524] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [10891.702507] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10895.453526] Lustre: 263558:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10902.501683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10902.508438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10902.515414] Lustre: Skipped 4 previous similar messages [10902.526355] Lustre: Skipped 3 previous similar messages [10907.621794] LustreError: 260541:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10907.652001] LustreError: 260541:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10907.750187] Lustre: server umount lustre-MDT0000 complete [10911.954513] LustreError: 260526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782619600 with bad export cookie 15242988958324506501 [10911.956787] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10911.962335] LustreError: 260526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10912.374693] Lustre: server umount lustre-MDT0001 complete [10927.758613] Lustre: server umount lustre-OST0000 complete [10928.930319] Lustre: 250521:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782619601/real 1782619601] req@ffff8e9c417a1f80 x1869211749187072/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782619617 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10928.971336] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10928.989901] Lustre: Skipped 1 previous similar message [10931.948478] Lustre: server umount lustre-OST0001 complete [10952.355282] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 00:07:18 (1782619638) [10972.672987] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [10984.485543] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [11027.903900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11028.177418] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11028.212392] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11028.318547] Lustre: lustre-MDT0000: new disk, initializing [11028.434555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11028.444856] Lustre: Skipped 1 previous similar message [11028.464841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11034.219910] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11046.293818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11046.492547] Lustre: 267812:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11046.589491] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11046.597659] Lustre: Skipped 1 previous similar message [11046.710853] Lustre: lustre-MDT0001: new disk, initializing [11046.910549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11046.921260] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11052.159957] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11062.754159] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11068.977711] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11076.968052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11077.349379] Lustre: lustre-OST0000: new disk, initializing [11077.358282] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11077.372241] Lustre: 269781:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11078.777891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11078.814388] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11078.946107] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11084.274780] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11094.873808] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11100.532358] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11100.929623] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11106.695711] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11106.919947] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11109.868156] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11109.882043] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11109.915688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11109.923692] Lustre: Skipped 1 previous similar message [11115.660802] Lustre: server umount lustre-OST0000 complete [11123.680775] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11123.701512] LustreError: Skipped 1 previous similar message [11123.707343] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11123.717550] Lustre: Skipped 1 previous similar message [11123.721477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11123.725527] Lustre: Skipped 2 previous similar messages [11127.236442] Lustre: server umount lustre-MDT0000 complete [11130.341405] LustreError: 268689:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11130.365517] LustreError: 268689:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11130.976653] LustreError: 267806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782619819 with bad export cookie 15242988958324508594 [11130.989397] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11130.991730] LustreError: 267806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11131.359789] Lustre: server umount lustre-MDT0001 complete [11151.540696] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [11162.397378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11162.895991] LustreError: 272719:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11162.970741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11162.986071] Lustre: Skipped 2 previous similar messages [11167.790952] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11177.403388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11182.401973] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11185.115536] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11191.482302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11198.230762] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11207.569337] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11207.779806] Lustre: lustre-OST0001: new disk, initializing [11207.786725] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11207.794561] Lustre: 275029:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11209.452206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11209.457992] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11209.552651] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11214.035543] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11223.754881] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11227.317641] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11232.618759] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [11232.685172] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11239.399861] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11239.408729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11239.420245] Lustre: Skipped 1 previous similar message [11239.429076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11239.433895] Lustre: Skipped 2 previous similar messages [11245.764642] Lustre: server umount lustre-MDT0000 complete [11249.822551] LustreError: 272701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782619938 with bad export cookie 15242988958324509476 [11249.842239] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11249.844528] LustreError: 272701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11250.237478] Lustre: server umount lustre-MDT0001 complete [11264.736148] Lustre: server umount lustre-OST0000 complete [11266.594952] Lustre: 250522:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782619939/real 1782619939] req@ffff8e9c4228dc00 x1869211749298304/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782619955 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11269.173754] Lustre: server umount lustre-OST0001 complete [11287.768236] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [11299.709253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11300.112134] LustreError: 278084:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11300.134854] LustreError: 278084:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11300.205188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11300.210229] Lustre: Skipped 3 previous similar messages [11304.938934] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11314.172966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11319.831057] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11323.484767] Lustre: 279225:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11331.366549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11338.594846] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11345.154427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11348.315989] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11353.621626] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11355.492855] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11364.297654] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11368.611864] Lustre: 281103:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11374.449553] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11374.472425] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11385.889411] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 00:14:32 (1782620072) [11401.703305] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11401.716048] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11401.726914] Lustre: Skipped 4 previous similar messages [11401.735979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11401.742865] Lustre: Skipped 8 previous similar messages [11407.355875] Lustre: server umount lustre-MDT0000 complete [11411.456023] LustreError: 278066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782620100 with bad export cookie 15242988958324516245 [11411.471907] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11417.897586] Lustre: server umount lustre-MDT0001 complete [11435.999422] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11436.267935] Lustre: server umount lustre-OST0000 complete [11440.748479] Lustre: server umount lustre-OST0001 complete [11448.203653] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [11459.496175] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [11506.600534] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [11517.257794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11517.523627] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11517.575506] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11517.677719] Lustre: lustre-MDT0000: new disk, initializing [11517.838551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11522.513640] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11533.997944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11534.113247] Lustre: 285348:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11534.385358] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11539.111753] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11550.387058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11550.680240] Lustre: 286980:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11552.182501] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11556.556501] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11567.350080] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11567.580423] Lustre: 287853:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11567.684070] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11567.693721] Lustre: Skipped 6 previous similar messages [11569.218094] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11569.228122] Lustre: Skipped 1 previous similar message [11569.333698] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11574.186449] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11584.827605] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11588.614839] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11588.623790] Lustre: Skipped 1 previous similar message [11603.809541] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 00:18:10 (1782620290) [11605.984173] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11605.993253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11606.007697] Lustre: Skipped 5 previous similar messages [11606.018200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11606.021201] Lustre: Skipped 8 previous similar messages [11612.338162] Lustre: server umount lustre-MDT0000 complete [11615.714622] LustreError: 286224:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11615.748619] LustreError: 286224:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 16 previous similar messages [11615.922383] LustreError: 285339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782620304 with bad export cookie 15242988958324523651 [11615.926585] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11615.937439] LustreError: 285339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11616.360345] Lustre: server umount lustre-MDT0001 complete [11630.124227] Lustre: server umount lustre-OST0000 complete [11643.362348] Lustre: server umount lustre-OST0001 complete [11652.820310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11658.109587] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11666.969566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11671.449890] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11679.015608] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11684.439493] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11690.828805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11697.053912] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11703.857801] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11718.287547] loop1: detected capacity change from 0 to 204800000 [11718.599603] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11718.613266] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11718.630925] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11723.762599] loop1: detected capacity change from 0 to 204800000 [11729.054027] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11729.144669] LustreError: 290090:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs28922: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11729.155352] LustreError: 290090:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs28922 [11729.160897] LustreError: 290090:0:(mgs_handler.c:517:mgs_target_reg()) fs28922-MDT0001: Error checking target index: rc = -13 [11729.169469] LustreError: fs28922-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11729.176083] LustreError: 293819:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11729.188966] LustreError: 293819:0:(tgt_mount.c:2129:server_put_super()) no obd fs28922-MDT0001 [11729.194358] LustreError: 293819:0:(tgt_mount.c:116:server_deregister_mount()) fs28922-MDT0001 not registered [11729.236111] Lustre: server umount fs28922-MDT0001 complete [11729.239453] LustreError: 293819:0:(super25.c:185:lustre_fill_super()) llite: Unable to mount : rc = -13 [11737.720810] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11737.867419] Lustre: Setting parameter fs28922-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs28922-MDT0001 [11737.927502] Lustre: srv-fs28922-MDT0001: No data found on store. Initialize space. [11737.931194] Lustre: Skipped 4 previous similar messages [11738.059173] Lustre: fs28922-MDT0001: new disk, initializing [11738.066322] Lustre: Skipped 3 previous similar messages [11738.165631] Lustre: srv-fs28922-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11738.173168] Lustre: 294287:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs28922-MDT0001-mdtlov: can't create new llog: rc = -115 [11738.185650] LustreError: 294287:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs28922-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11742.335651] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11749.095142] Lustre: server umount fs28922-MDT0001 complete [11761.777420] loop1: detected capacity change from 0 to 204800000 [11767.746791] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11768.136353] Lustre: 295854:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs28922-MDT0001-mdtlov: can't create new llog: rc = -115 [11768.149233] LustreError: 295854:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs28922-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11772.652638] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11775.472666] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11777.526167] Lustre: server umount fs28922-MDT0001 complete [11790.307271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11790.321325] Lustre: Skipped 3 previous similar messages [11790.325876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11790.330626] Lustre: Skipped 4 previous similar messages [11793.305578] Lustre: server umount lustre-MDT0000 complete [11801.595822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11801.758090] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11805.807146] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11807.217962] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11812.474172] loop1: detected capacity change from 0 to 204800000 [11817.390236] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11817.775561] Lustre: 298302:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs28922-MDT0001-mdtlov: can't create new llog: rc = -115 [11817.783770] LustreError: 298302:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs28922-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11822.143284] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11826.939493] Lustre: server umount fs28922-MDT0001 complete [11848.170189] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11848.184573] LustreError: Skipped 2 previous similar messages [11853.556645] Lustre: server umount lustre-OST0000 complete [11863.749754] Lustre: server umount lustre-MDT0000 complete [11867.439477] LustreError: 290082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782620556 with bad export cookie 15242988958324526332 [11867.446717] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11867.455719] LustreError: 290082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11867.799127] Lustre: server umount lustre-MDT0001 complete [11879.078501] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [11881.951482] Key type lgssc unregistered [11882.224197] LNet: 300647:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11882.231322] LNetError: 300647:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11882.252969] LNet: Removed LNI 192.168.204.160@tcp [11882.959434] Key type .llcrypt unregistered [11882.961522] Key type ._llcrypt unregistered [11900.246977] Key type ._llcrypt registered [11900.249525] Key type .llcrypt registered [11900.344667] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [11901.079176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11901.124248] alg: No test for adler32 (adler32-zlib) [11902.295132] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [11902.553825] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [11904.247522] Key type lgssc registered [11905.356131] Lustre: Echo OBD driver; http://www.lustre.org/ [11912.928223] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11912.945037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11914.463925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11918.979612] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11924.965779] LustreError: 301922:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11924.983733] LustreError: 301922:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11927.080807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11927.405874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11931.343488] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11938.069964] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11943.150991] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11949.154908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11949.604668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11952.683613] LustreError: 303751:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11955.081533] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11957.752692] LustreError: 303943:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11959.778113] LustreError: 303942:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11959.804342] LustreError: 303942:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11962.441719] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11964.907637] LustreError: 303942:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11964.929412] LustreError: 303942:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11975.144156] LustreError: 303750:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11975.158393] LustreError: 303750:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [11977.583528] loop1: detected capacity change from 0 to 204800000 [11977.760438] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11977.770221] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11977.785261] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11981.990441] loop1: detected capacity change from 0 to 204800000 [11986.965811] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11987.106081] Lustre: fs4583-OST0000: new disk, initializing [11987.112478] Lustre: srv-fs4583-OST0000: No data found on store. Initialize space. [11987.120827] Lustre: 305654:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11987.183796] Lustre: fs4583-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11992.971207] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [11995.622567] LustreError: 303751:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11995.647507] LustreError: 303751:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [11999.791392] Lustre: server umount fs4583-OST0000 complete [12010.351433] loop1: detected capacity change from 0 to 204800000 [12014.960309] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12015.101284] Lustre: fs4583-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12019.823724] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [12026.971153] loop2: detected capacity change from 0 to 204800000 [12027.140313] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12027.154187] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12027.162615] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12031.268433] loop2: detected capacity change from 0 to 204800000 [12031.467374] LustreError: 303750:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12031.479295] LustreError: 303750:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 20 previous similar messages [12035.424673] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12035.514512] LustreError: fs4583-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [12035.522821] LustreError: fs4583-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [12035.542362] LustreError: 308692:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [12035.558552] LustreError: 308692:0:(tgt_mount.c:2129:server_put_super()) no obd fs4583-OST0001 [12035.571414] LustreError: 308692:0:(tgt_mount.c:116:server_deregister_mount()) fs4583-OST0001 not registered [12035.626153] Lustre: server umount fs4583-OST0001 complete [12035.631991] LustreError: 308692:0:(super25.c:185:lustre_fill_super()) llite: Unable to mount : rc = -13 [12036.584905] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12041.707518] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12045.379721] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12045.600238] Lustre: fs4583-OST0001: new disk, initializing [12045.607905] Lustre: srv-fs4583-OST0001: No data found on store. Initialize space. [12045.621031] Lustre: 309242:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12045.760209] Lustre: fs4583-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12050.728221] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [12057.621419] Lustre: server umount fs4583-OST0000 complete [12065.530257] Lustre: server umount fs4583-OST0001 complete [12076.425773] loop1: detected capacity change from 0 to 204800000 [12081.134530] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12081.382176] Lustre: fs4583-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12086.762439] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [12092.558468] loop2: detected capacity change from 0 to 204800000 [12097.660508] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12098.025198] LustreError: 303750:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12098.039670] LustreError: 303750:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 38 previous similar messages [12102.944632] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [12109.733396] Lustre: server umount fs4583-OST0000 complete [12117.585787] Lustre: server umount fs4583-OST0001 complete [12128.740593] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12128.744126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12128.749839] Lustre: Skipped 1 previous similar message [12128.750787] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12128.768353] Lustre: Skipped 1 previous similar message [12133.861023] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12133.871571] Lustre: Skipped 1 previous similar message [12134.523504] Lustre: server umount lustre-OST0000 complete [12137.440676] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12137.448967] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12137.461611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12138.977369] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12143.604883] Lustre: server umount lustre-MDT0000 complete [12146.436512] LustreError: 309989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782620835 with bad export cookie 10459772927806742624 [12146.439750] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12146.446503] LustreError: 309989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12146.733634] Lustre: server umount lustre-MDT0001 complete [12155.587151] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [12158.089401] Key type lgssc unregistered [12158.433220] LNet: 315164:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12158.437991] LNetError: 315164:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12158.455420] LNet: Removed LNI 192.168.204.160@tcp [12159.293154] Key type .llcrypt unregistered [12159.297278] Key type ._llcrypt unregistered [12184.705279] Key type ._llcrypt registered [12184.707632] Key type .llcrypt registered [12184.795530] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [12197.149607] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [12197.969960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12197.984072] alg: No test for adler32 (adler32-zlib) [12198.940527] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [12199.129661] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [12200.759752] Key type lgssc registered [12201.584945] Lustre: Echo OBD driver; http://www.lustre.org/ [12227.089637] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12227.113369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12228.291658] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12228.324558] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12228.420670] Lustre: lustre-MDT0000: new disk, initializing [12228.498068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12228.525989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12231.606597] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [12239.748904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12239.831147] Lustre: 319001:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12239.885949] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12239.890186] Lustre: Skipped 1 previous similar message [12239.941974] Lustre: lustre-MDT0001: new disk, initializing [12240.027415] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12240.056478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12240.060676] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12243.491602] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [12251.098318] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12255.052748] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12260.377883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12260.556859] Lustre: lustre-OST0000: new disk, initializing [12260.560027] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12260.564267] Lustre: 320970:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12260.624942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12264.616263] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [12269.103269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12269.107509] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12269.211600] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12271.759877] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12273.178664] Lustre: Mounted lustre-client [12273.941397] Lustre: Unmounted lustre-client [12283.362267] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12283.376034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12283.385546] Lustre: Skipped 1 previous similar message [12287.519332] Lustre: server umount lustre-OST0000 complete [12291.553159] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12291.557872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12291.565229] Lustre: Skipped 1 previous similar message [12291.569535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12293.601133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12293.602490] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12296.392206] Lustre: server umount lustre-MDT0000 complete [12299.245111] LustreError: 318994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782620988 with bad export cookie 9395399290130098255 [12299.255139] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12299.255404] LustreError: 318994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12299.459949] Lustre: server umount lustre-MDT0001 complete [12306.685962] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [12308.921425] Key type lgssc unregistered [12329.632898] LNet: 323127:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12329.639041] LNetError: 323127:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12404.395318] LNet: Removed LNI 192.168.204.160@tcp [12405.052852] Key type .llcrypt unregistered [12405.056352] Key type ._llcrypt unregistered [12417.227647] Key type ._llcrypt registered [12417.229625] Key type .llcrypt registered [12417.323593] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [12429.508304] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [12430.096050] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12430.115775] alg: No test for adler32 (adler32-zlib) [12431.095624] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [12431.270107] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [12432.919192] Key type lgssc registered [12433.762643] Lustre: Echo OBD driver; http://www.lustre.org/ [12455.443163] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 00:32:22 (1782621142) [12459.609463] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12459.626327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12460.794572] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12460.819301] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12460.868512] Lustre: lustre-MDT0000: new disk, initializing [12460.931236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12460.947181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12463.600502] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [12471.569291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12471.662084] Lustre: 326344:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12471.711425] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12471.719587] Lustre: Skipped 1 previous similar message [12471.778135] Lustre: lustre-MDT0001: new disk, initializing [12471.835425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12471.863532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12471.873659] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12474.830836] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [12481.060225] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12484.402725] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12488.331722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12488.490702] Lustre: lustre-OST0000: new disk, initializing [12488.497887] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12488.503189] Lustre: 328311:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12488.559412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12488.856466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12488.866655] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12488.945641] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12491.957371] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [12497.693740] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12509.666274] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12509.666588] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12509.668578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12509.671980] LustreError: Skipped 1 previous similar message [12509.679456] Lustre: Skipped 1 previous similar message [12512.893629] Lustre: server umount lustre-OST0000 complete [12517.856423] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12517.864055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12517.876959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12517.880157] Lustre: Skipped 1 previous similar message [12519.906800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12519.907317] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12521.280690] Lustre: server umount lustre-MDT0000 complete [12523.513578] LustreError: 326335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782621212 with bad export cookie 4130977580686221172 [12523.516403] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12523.520383] LustreError: 326335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12523.730709] Lustre: server umount lustre-MDT0001 complete [12529.872398] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [12531.499603] Key type lgssc unregistered [12531.679213] LNet: 330216:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12531.685202] LNetError: 330216:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12532.710145] LNet: Removed LNI 192.168.204.160@tcp [12533.175685] Key type .llcrypt unregistered [12533.177854] Key type ._llcrypt unregistered [12544.723183] Key type ._llcrypt registered [12544.725047] Key type .llcrypt registered [12544.781824] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [12545.363606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12545.388030] alg: No test for adler32 (adler32-zlib) [12546.320608] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [12546.472965] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [12548.095172] Key type lgssc registered [12548.830061] Lustre: Echo OBD driver; http://www.lustre.org/ [12553.232152] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12553.251041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12554.484279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12557.001360] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [12562.374125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12562.601365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12565.497482] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [12570.097388] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12573.495258] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12577.736640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12577.938098] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12581.342627] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [12586.170747] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12591.788702] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12592.597737] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12597.730342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12597.733179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12597.738878] Lustre: Skipped 1 previous similar message [12600.703514] Lustre: server umount lustre-MDT0000 complete [12602.827563] LustreError: 331450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782621291 with bad export cookie 11164303999402359572 [12602.829322] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12602.839417] LustreError: 331450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12602.850273] LustreError: 331466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12602.850288] LustreError: 331466:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12602.850397] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12602.850403] Lustre: Skipped 1 previous similar message [12602.851358] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12602.886017] Lustre: Skipped 3 previous similar messages [12607.968075] LustreError: 332902:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12607.968388] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12607.983980] LustreError: 332902:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12607.998592] Lustre: Skipped 1 previous similar message [12612.068511] LustreError: 331466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12612.068580] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12612.079588] LustreError: 331466:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12612.082780] Lustre: Skipped 1 previous similar message [12617.183144] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12617.186291] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12617.186298] LustreError: 331467:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12617.186318] LustreError: 331467:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12617.202872] Lustre: Skipped 1 previous similar message [12617.337890] Lustre: server umount lustre-MDT0001 complete [12619.710196] Lustre: server umount lustre-OST0000 complete [12624.975074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12627.432920] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [12629.105843] Lustre: 335646:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.160@tcp [12630.380961] Lustre: server umount MGS complete [12635.351975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12635.629456] LustreError: 336217:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12635.675731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12638.315597] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [12643.608644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12643.817622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12646.306631] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [12651.231202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12654.623640] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [12658.572740] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12659.409354] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12662.220409] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12667.265991] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [12670.954412] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12670.955646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12670.962126] Lustre: Skipped 1 previous similar message [12670.962663] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12670.973922] Lustre: Skipped 1 previous similar message [12675.434445] Lustre: server umount lustre-OST0000 complete [12679.648835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12679.659068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12683.788915] Lustre: server umount lustre-MDT0000 complete [12686.248822] LustreError: 336199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782621375 with bad export cookie 11164303999402360398 [12686.254937] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12686.255349] LustreError: 336199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12686.463391] Lustre: server umount lustre-MDT0001 complete [12692.623843] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [12694.223545] Key type lgssc unregistered [12694.429250] LNet: 339651:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12694.437486] LNetError: 339651:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12694.456689] LNet: Removed LNI 192.168.204.160@tcp [12694.940990] Key type .llcrypt unregistered [12694.943477] Key type ._llcrypt unregistered [12705.448054] Key type ._llcrypt registered [12705.450083] Key type .llcrypt registered [12705.536494] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [12714.802303] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [12715.253735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12715.319811] alg: No test for adler32 (adler32-zlib) [12716.244298] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [12716.387639] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [12717.999198] Key type lgssc registered [12718.688707] Lustre: Echo OBD driver; http://www.lustre.org/ [12738.327783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12738.349548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12739.467676] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12739.486628] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12739.527668] Lustre: lustre-MDT0000: new disk, initializing [12739.566373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12739.578604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12741.608603] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [12748.126780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12748.216811] Lustre: 342789:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12748.277352] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12748.281925] Lustre: Skipped 1 previous similar message [12748.346249] Lustre: lustre-MDT0001: new disk, initializing [12748.416659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12748.441526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12748.451975] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12750.944727] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [12756.074657] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12758.494561] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12761.812601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12761.965893] Lustre: lustre-OST0000: new disk, initializing [12761.969210] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12761.976348] Lustre: 344753:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12762.040480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12765.021466] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [12769.319074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12769.325172] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12769.376217] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12770.188340] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12772.572107] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12772.675445] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12775.108053] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12775.223425] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12779.488501] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12779.490643] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12779.493501] LustreError: Skipped 1 previous similar message [12779.499061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12779.504124] Lustre: Skipped 1 previous similar message [12782.442520] Lustre: server umount lustre-OST0000 complete [12784.609621] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12784.609813] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12784.613502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12784.633181] Lustre: Skipped 2 previous similar messages [12789.729016] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12789.733156] Lustre: Skipped 2 previous similar messages [12790.617172] Lustre: server umount lustre-MDT0000 complete [12792.564754] LustreError: 342779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782621481 with bad export cookie 9027009639214615172 [12792.567552] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12792.569751] LustreError: 342779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12792.770578] Lustre: server umount lustre-MDT0001 complete [12797.634418] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 00:38:04 (1782621484) [12806.179303] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [12810.799755] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [12829.280490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12829.392100] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12829.411392] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12829.457352] Lustre: lustre-MDT0000: new disk, initializing [12829.500108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12829.513786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12831.658386] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [12837.495318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12837.579979] Lustre: 349505:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12837.619792] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12837.624154] Lustre: Skipped 1 previous similar message [12837.668830] Lustre: lustre-MDT0001: new disk, initializing [12837.718024] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12837.742359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12837.750722] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12839.668370] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [12845.091138] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12848.144973] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12851.667448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12851.822712] Lustre: lustre-OST0000: new disk, initializing [12851.826430] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12851.830389] Lustre: 351470:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12851.875868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12855.236637] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [12859.402860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12859.412168] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12859.473691] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12860.871514] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12884.960659] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12884.964469] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12884.969505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12884.972826] Lustre: Skipped 1 previous similar message [12886.265224] Lustre: server umount lustre-OST0000 complete [12889.056358] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12889.061147] LustreError: Skipped 1 previous similar message [12889.064803] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12889.072907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12889.079795] Lustre: Skipped 1 previous similar message [12894.556374] Lustre: server umount lustre-MDT0000 complete [12896.808376] LustreError: 349495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782621585 with bad export cookie 9027009639214616033 [12896.811549] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12896.817082] LustreError: 349495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12897.004923] Lustre: server umount lustre-MDT0001 complete [12903.241532] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [12904.764665] Key type lgssc unregistered [12904.951485] LNet: 353619:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12904.956852] LNetError: 353619:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12904.970664] LNet: Removed LNI 192.168.204.160@tcp [12905.414898] Key type .llcrypt unregistered [12905.416615] Key type ._llcrypt unregistered [12957.756915] Key type ._llcrypt registered [12957.758214] Key type .llcrypt registered [12957.820597] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [12957.995337] Key type .llcrypt unregistered [12957.999792] Key type ._llcrypt unregistered [12998.643158] Key type ._llcrypt registered [12998.644329] Key type .llcrypt registered [12998.697358] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [12998.817558] Key type .llcrypt unregistered [12998.819668] Key type ._llcrypt unregistered [13012.870876] Key type ._llcrypt registered [13012.872510] Key type .llcrypt registered [13012.939989] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [13013.083379] Key type .llcrypt unregistered [13013.084720] Key type ._llcrypt unregistered [13028.160976] Key type ._llcrypt registered [13028.162799] Key type .llcrypt registered [13028.220509] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [13028.370153] Key type .llcrypt unregistered [13028.371936] Key type ._llcrypt unregistered [13065.345225] Key type ._llcrypt registered [13065.347462] Key type .llcrypt registered [13065.412352] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [13065.579275] Key type .llcrypt unregistered [13065.581681] Key type ._llcrypt unregistered [13081.429878] Key type ._llcrypt registered [13081.431358] Key type .llcrypt registered [13081.489245] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [13081.649694] Key type .llcrypt unregistered [13081.652994] Key type ._llcrypt unregistered [13096.245707] Key type ._llcrypt registered [13096.247110] Key type .llcrypt registered [13096.298785] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [13096.421497] Key type .llcrypt unregistered [13096.423050] Key type ._llcrypt unregistered [13110.412204] Key type ._llcrypt registered [13110.415525] Key type .llcrypt registered [13110.492977] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [13110.649455] Key type .llcrypt unregistered [13110.652512] Key type ._llcrypt unregistered [13125.428537] Key type ._llcrypt registered [13125.430143] Key type .llcrypt registered [13125.495719] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [13125.653516] Key type .llcrypt unregistered [13125.655383] Key type ._llcrypt unregistered [13140.759422] Key type ._llcrypt registered [13140.761475] Key type .llcrypt registered [13140.813531] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [13140.959606] Key type .llcrypt unregistered [13140.962779] Key type ._llcrypt unregistered [13172.598723] Key type ._llcrypt registered [13172.600833] Key type .llcrypt registered [13172.660565] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [13172.813640] Key type .llcrypt unregistered [13172.816673] Key type ._llcrypt unregistered [13187.831644] Key type ._llcrypt registered [13187.833419] Key type .llcrypt registered [13187.896565] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [13188.062455] Key type .llcrypt unregistered [13188.065769] Key type ._llcrypt unregistered [13202.947040] Key type ._llcrypt registered [13202.948538] Key type .llcrypt registered [13202.997829] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [13203.133263] Key type .llcrypt unregistered [13203.134916] Key type ._llcrypt unregistered [13229.553254] Key type ._llcrypt registered [13229.555284] Key type .llcrypt registered [13229.617233] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [13237.880609] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [13238.393529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13238.442336] alg: No test for adler32 (adler32-zlib) [13239.354206] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [13239.508881] LNet: Added LNI 192.168.204.160@tcp [8/256/0/180] [13241.111320] Key type lgssc registered [13241.786461] Lustre: Echo OBD driver; http://www.lustre.org/ [13259.448186] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13259.463867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13260.593853] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13260.613744] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13260.657361] Lustre: tst250fs-MDT0000: new disk, initializing [13260.703198] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13260.715472] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13262.930555] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [13268.870754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13268.929957] Lustre: 373537:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13268.969941] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13268.973569] Lustre: Skipped 1 previous similar message [13269.018491] Lustre: tst250fs-MDT0001: new disk, initializing [13269.082938] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13269.105291] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13269.110370] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13271.291655] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [13274.420140] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13277.718580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13277.854122] Lustre: tst250fs-OST0000: new disk, initializing [13277.858204] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13277.862614] Lustre: 375165:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13277.911112] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13280.582384] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [13280.767683] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13280.772930] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13280.844993] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13286.238170] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13286.334048] Lustre: tst250fs-OST0001: new disk, initializing [13286.336595] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13286.340886] Lustre: 376038:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13286.394121] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13287.958425] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13287.964480] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13288.002685] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13289.287286] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_default_debug -1 all [13292.758402] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13308.389775] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13308.393230] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13308.402064] Lustre: Skipped 3 previous similar messages [13308.405654] Lustre: Skipped 2 previous similar messages [13311.898351] Lustre: server umount tst250fs-MDT0000 complete [13313.858208] LustreError: 374332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782622002 with bad export cookie 1053063662996799497 [13313.865064] LustreError: MGC192.168.204.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13313.872375] LustreError: 374332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13314.036398] Lustre: server umount tst250fs-MDT0001 complete [13326.858415] Lustre: server umount tst250fs-OST0000 complete [13338.860607] Lustre: server umount tst250fs-OST0001 complete [13347.749443] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [13351.630804] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [13368.518418] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 00:47:35 (1782622055) [13369.380400] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13377.166171] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing set_hostid [13381.581899] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing load_modules_local [13396.023465] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13113 sec ========= 00:48:03 (1782622083) [13396.794913] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 00:48:04 (1782622084) === [13398.490868] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 00:48:05 (1782622085) === [13415.268539] Lustre: DEBUG MARKER: oleg460-server.virtnet: executing unload_modules_local [13416.414423] Key type lgssc unregistered [13416.544372] LNet: 385156:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13416.548653] LNetError: 385156:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13416.556444] LNet: Removed LNI 192.168.204.160@tcp [13416.937250] Key type .llcrypt unregistered [13416.939562] Key type ._llcrypt unregistered