[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 578706333 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.002335] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.006749] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.007024] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.008016] pid_max: default: 32768 minimum: 301 [ 0.009274] LSM: Security Framework initializing [ 0.010060] Yama: becoming mindful. [ 0.011039] SELinux: Initializing. [ 0.013008] *** VALIDATE selinux *** [ 0.022421] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027601] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028161] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030118] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032131] *** VALIDATE tmpfs *** [ 0.033499] *** VALIDATE proc *** [ 0.034239] *** VALIDATE cgroup *** [ 0.035000] *** VALIDATE cgroup2 *** [ 0.035000] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035192] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036017] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037045] Spectre V2 : User space: Vulnerable [ 0.038010] Speculative Store Bypass: Vulnerable [ 0.041440] debug: unmapping init [mem 0xffffffff8a459000-0xffffffff8a460fff] [ 0.043150] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044690] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045024] ... version: 2 [ 0.046011] ... bit width: 48 [ 0.047009] ... generic registers: 4 [ 0.048008] ... value mask: 0000ffffffffffff [ 0.049008] ... max period: 00007fffffffffff [ 0.050008] ... fixed-purpose events: 3 [ 0.051007] ... event mask: 000000070000000f [ 0.052308] rcu: Hierarchical SRCU implementation. [ 0.054461] smp: Bringing up secondary CPUs ... [ 0.055484] x86: Booting SMP configuration: [ 0.056015] .... node #0, CPUs: #1 #2 #3 [ 0.060572] smp: Brought up 1 node, 4 CPUs [ 0.062017] smpboot: Max logical packages: 1 [ 0.063014] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.160029] node 0 deferred pages initialised in 93ms [ 0.166166] devtmpfs: initialized [ 0.167219] x86/mm: Memory block size: 128MB [ 0.169000] gcov: version magic: 0x41383552 [ 0.172374] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.175093] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.178348] pinctrl core: initialized pinctrl subsystem [ 0.181163] [ 0.182006] ************************************************************* [ 0.184013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.186013] ** ** [ 0.187016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.190015] ** ** [ 0.192013] ** This means that this kernel is built to expose internal ** [ 0.195016] ** IOMMU data structures, which may compromise security on ** [ 0.197013] ** your system. ** [ 0.200061] ** ** [ 0.203016] ** If you see this message and you are not debugging the ** [ 0.206018] ** kernel, report this immediately to your vendor! ** [ 0.208015] ** ** [ 0.211019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.214017] ************************************************************* [ 0.219245] NET: Registered protocol family 16 [ 0.220485] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.221178] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.224073] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.229148] cpuidle: using governor menu [ 0.246042] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.248609] PCI: Using configuration type 1 for base access [ 0.252140] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.265292] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.267087] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.271059] cryptd: max_cpu_qlen set to 1000 [ 0.273222] ACPI: Added _OSI(Module Device) [ 0.274027] ACPI: Added _OSI(Processor Device) [ 0.277041] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.282031] ACPI: Added _OSI(Processor Aggregator Device) [ 0.290800] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.296269] ACPI: Interpreter enabled [ 0.297000] ACPI: PM: (supports S0 S3 S4 S5) [ 0.297000] ACPI: Using IOAPIC for interrupt routing [ 0.299112] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.305144] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.319541] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.323050] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.326023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.332113] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.338236] acpiphp: Slot [2] registered [ 0.341156] acpiphp: Slot [3] registered [ 0.342080] acpiphp: Slot [4] registered [ 0.343067] acpiphp: Slot [5] registered [ 0.345141] acpiphp: Slot [6] registered [ 0.347086] acpiphp: Slot [7] registered [ 0.349103] acpiphp: Slot [8] registered [ 0.351093] acpiphp: Slot [9] registered [ 0.353086] acpiphp: Slot [10] registered [ 0.356098] acpiphp: Slot [11] registered [ 0.358270] acpiphp: Slot [12] registered [ 0.359076] acpiphp: Slot [13] registered [ 0.360066] acpiphp: Slot [14] registered [ 0.362060] acpiphp: Slot [15] registered [ 0.363052] acpiphp: Slot [16] registered [ 0.364061] acpiphp: Slot [17] registered [ 0.365069] acpiphp: Slot [18] registered [ 0.366060] acpiphp: Slot [19] registered [ 0.368057] acpiphp: Slot [20] registered [ 0.369074] acpiphp: Slot [21] registered [ 0.370000] acpiphp: Slot [22] registered [ 0.370000] acpiphp: Slot [23] registered [ 0.372071] acpiphp: Slot [24] registered [ 0.374092] acpiphp: Slot [25] registered [ 0.375068] acpiphp: Slot [26] registered [ 0.376078] acpiphp: Slot [27] registered [ 0.377000] acpiphp: Slot [28] registered [ 0.377000] acpiphp: Slot [29] registered [ 0.379076] acpiphp: Slot [30] registered [ 0.380079] acpiphp: Slot [31] registered [ 0.382051] PCI host bridge to bus 0000:00 [ 0.383015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.386019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.389021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.392019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.396029] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.397017] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.398152] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.402524] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.407753] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.420033] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.427925] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.431014] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.435020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.439031] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.444568] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.447733] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.450038] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.452584] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.456011] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.480020] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.485978] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.492096] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.500015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.504015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.517016] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.529930] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.538015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.544015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.557018] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.570129] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.577028] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.589013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.606016] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.617917] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.627014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.634024] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.655015] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.663334] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.668012] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.672014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.683014] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.690431] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.694012] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.699013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.706017] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.713761] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.716333] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.718447] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.720320] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.722155] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.726209] iommu: Default domain type: Passthrough [ 0.727000] SCSI subsystem initialized [ 0.727121] ACPI: bus type USB registered [ 0.728000] usbcore: registered new interface driver usbfs [ 0.729074] usbcore: registered new interface driver hub [ 0.730055] usbcore: registered new device driver usb [ 0.732145] pps_core: LinuxPPS API ver. 1 registered [ 0.734007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.737049] PTP clock support registered [ 0.738187] EDAC MC: Ver: 3.0.0 [ 0.739816] PCI: Using ACPI for IRQ routing [ 0.740924] NetLabel: Initializing [ 0.741000] NetLabel: domain hash size = 128 [ 0.742011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.745132] NetLabel: unlabeled traffic allowed by default [ 0.747321] vgaarb: loaded [ 0.749184] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.750000] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.759896] clocksource: Switched to clocksource kvm-clock [ 0.874311] VFS: Disk quotas dquot_6.6.0 [ 0.875653] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.878093] *** VALIDATE ramfs *** [ 0.879283] *** VALIDATE hugetlbfs *** [ 0.880704] pnp: PnP ACPI init [ 0.883165] pnp: PnP ACPI: found 6 devices [ 0.901107] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.904292] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.906340] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.908568] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.911012] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.913330] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.915741] NET: Registered protocol family 2 [ 0.917740] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.922679] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.925905] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.930553] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.934207] TCP: Hash tables configured (established 65536 bind 65536) [ 0.936702] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.939549] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.942160] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.945163] NET: Registered protocol family 1 [ 0.948648] RPC: Registered named UNIX socket transport module. [ 0.951505] RPC: Registered udp transport module. [ 0.953113] RPC: Registered tcp transport module. [ 0.954539] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.956682] NET: Registered protocol family 44 [ 0.958297] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.960282] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.962151] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.964486] PCI: CLS 0 bytes, default 64 [ 0.966872] Unpacking initramfs... [ 2.694859] debug: unmapping init [mem 0xffff8df27cc54000-0xffff8df27ffbffff] [ 2.700191] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.703712] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.706292] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 3.320887] Initialise system trusted keyrings [ 3.322185] Key type blacklist registered [ 3.324250] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.331817] zbud: loaded [ 3.334925] *** VALIDATE nfs *** [ 3.335835] *** VALIDATE nfs4 *** [ 3.336985] pstore: using deflate compression [ 3.340324] Platform Keyring initialized [ 3.510700] NET: Registered protocol family 38 [ 3.511986] Key type asymmetric registered [ 3.513060] Asymmetric key parser 'x509' registered [ 3.514711] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.517995] io scheduler mq-deadline registered [ 3.519636] io scheduler kyber registered [ 3.520961] io scheduler bfq registered [ 3.522459] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.525617] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.534909] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.541309] ACPI: Power Button [PWRF] [ 3.658034] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.795246] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.041858] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.142944] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.340959] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.371950] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.403301] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.410541] Non-volatile memory driver v1.3 [ 4.412190] Linux agpgart interface v0.103 [ 4.448995] virtio_blk virtio1: [vda] 133160 512-byte logical blocks (68.2 MB/65.0 MiB) [ 4.452089] vda: detected capacity change from 0 to 68177920 [ 4.476887] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.479788] vdb: detected capacity change from 0 to 1073741824 [ 4.499368] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.501454] vdc: detected capacity change from 0 to 2621440000 [ 4.527060] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.533126] vdd: detected capacity change from 0 to 2621440000 [ 4.546788] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.550195] vde: detected capacity change from 0 to 4294967296 [ 4.564900] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.569246] vdf: detected capacity change from 0 to 4294967296 [ 4.576003] libphy: Fixed MDIO Bus: probed [ 4.587522] usbcore: registered new interface driver usbserial_generic [ 4.592958] usbserial: USB Serial support registered for generic [ 4.595945] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.600553] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.602488] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.606777] mousedev: PS/2 mouse device common for all mice [ 4.612085] rtc_cmos 00:05: RTC can wake from S4 [ 4.619184] rtc_cmos 00:05: registered as rtc0 [ 4.620944] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.624155] intel_pstate: CPU model not supported [ 4.627419] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.634148] hid: raw HID events driver (C) Jiri Kosina [ 4.636088] usbcore: registered new interface driver usbhid [ 4.637734] usbhid: USB HID core driver [ 4.639549] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.644853] drop_monitor: Initializing network drop monitor service [ 4.652949] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.658214] Initializing XFRM netlink socket [ 4.660359] NET: Registered protocol family 10 [ 4.663834] Segment Routing with IPv6 [ 4.664786] NET: Registered protocol family 17 [ 4.667849] mpls_gso: MPLS GSO support [ 4.673649] RAS: Correctable Errors collector initialized. [ 4.675939] AVX version of gcm_enc/dec engaged. [ 4.677296] AES CTR mode by8 optimization enabled [ 4.783932] sched_clock: Marking stable (4783913145, 0)->(5884334964, -1100421819) [ 4.786832] registered taskstats version 1 [ 4.791717] Loading compiled-in X.509 certificates [ 4.793338] zswap: loaded using pool lzo/zbud [ 4.836452] Key type big_key registered [ 4.871304] Key type encrypted registered [ 4.872851] ima: No TPM chip found, activating TPM-bypass! [ 4.879112] ima: Allocated hash algorithm: sha1 [ 4.880387] ima: No architecture policies found [ 4.882348] evm: Initialising EVM extended attributes: [ 4.884713] evm: security.selinux [ 4.885771] evm: security.ima [ 4.887129] evm: security.capability [ 4.891965] evm: HMAC attrs: 0x1 [ 4.895721] rtc_cmos 00:05: setting system clock to 2025-09-24 17:04:32 UTC (1758733472) [ 4.902683] debug: unmapping init [mem 0xffffffff8b403000-0xffffffff8b5fffff] [ 4.909208] debug: unmapping init [mem 0xffffffff8a182000-0xffffffff8a458fff] [ 4.915107] Write protecting the kernel read-only data: 28672k [ 4.918556] debug: unmapping init [mem 0xffffffff88803000-0xffffffff889fffff] [ 4.920842] debug: unmapping init [mem 0xffffffff89114000-0xffffffff891fffff] [ 4.966589] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.976945] systemd[1]: Detected virtualization kvm. [ 4.978768] systemd[1]: Detected architecture x86-64. [ 4.980584] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.029120] systemd[1]: No hostname configured. [ 5.031069] systemd[1]: Set hostname to . [ 5.033128] random: systemd: uninitialized urandom read (16 bytes read) [ 5.035230] systemd[1]: Initializing machine ID from random generator. [ 5.332386] random: systemd: uninitialized urandom read (16 bytes read) [ 5.338842] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 5.344255] random: systemd: uninitialized urandom read (16 bytes read) [ 5.347298] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 5.353986] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.542205] device-mapper: uevent: version 1.0.3 [ 6.544961] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 8.086462] virtio_net virtio0 ens2: renamed from eth0 [ 8.119666] random: fast init done [ 8.765645] scsi host0: ata_piix [ 8.862394] scsi host1: ata_piix [ 8.867904] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.869991] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.410683] random: crng init done [ 13.411941] random: 7 urandom warning(s) missed due to ratelimiting [ 14.705281] dracut-initqueue[588]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 15.777510] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 17.920116] printk: systemd: 26 output lines suppressed due to ratelimiting [ 18.470256] SELinux: Disabled at runtime. [ 18.561785] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.578711] systemd[1]: Detected virtualization kvm. [ 18.581280] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 20.047616] systemd[1]: initrd-switch-root.service: Succeeded. [ 20.054418] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 20.065866] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 20.083251] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 20.091847] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 20.126496] systemd[1]: Starting Journal Service... Starting Journal Service... [ 20.138517] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 20.536726] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... [ OK ] Reached target Paths. Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 21.910861] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 23.342092] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 23.351694] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 24.650386] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 25.043153] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit)[ 31.225532] Key type dns_resolver registered [ *] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit)[ 32.375394] NFS: Registering the id_resolver key type [ 32.384819] Key type id_resolver registered [ 32.388289] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (12s / no limit) [ *** ] A start job is running for Configur…only root support (13s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg342-server login: [ 93.490919] libcfs: loading out-of-tree module taints kernel. [ 93.519743] Key type ._llcrypt registered [ 93.521374] Key type .llcrypt registered [ 93.600741] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 104.344286] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 105.051222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 105.059796] alg: No test for adler32 (adler32-zlib) [ 106.154339] Lustre: Lustre: Build Version: 2.16.58_51_g2f926bd [ 106.582144] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 108.232222] Key type lgssc registered [ 109.285106] Lustre: Echo OBD driver; http://www.lustre.org/ [ 119.258892] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 120.723367] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 125.608158] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 130.227064] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 134.892739] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 144.255588] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 151.796630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 151.841884] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 151.860202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 153.031536] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 153.062613] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 153.120310] Lustre: lustre-MDT0000: new disk, initializing [ 153.191235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 153.208781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 155.372659] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 163.435820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 163.488174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 163.543717] Lustre: 6496:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 163.578234] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 163.581629] Lustre: Skipped 1 previous similar message [ 163.655946] Lustre: lustre-MDT0001: new disk, initializing [ 163.699083] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 163.726273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 163.734272] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 165.607284] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 168.606065] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 173.670482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 173.713605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 173.830611] Lustre: lustre-OST0000: new disk, initializing [ 173.836445] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 173.875337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 175.551616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 175.554765] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 175.599240] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 176.384759] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 183.582407] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 183.631852] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 183.693889] Lustre: lustre-OST0001: new disk, initializing [ 183.696673] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 183.729951] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 184.568751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 184.573423] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 184.594786] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 186.505575] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 193.444367] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 197.657174] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 206.240273] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing check_logdir /tmp/testlogs/ [ 209.094834] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing yml_node [ 212.532431] Lustre: DEBUG MARKER: Client: 2.16.58.51 [ 214.246578] Lustre: DEBUG MARKER: MDS: 2.16.58.51 [ 215.599514] Lustre: DEBUG MARKER: OSS: 2.16.58.51 [ 216.467929] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Sep 24 13:08:02 EDT 2025 [ 227.927970] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 229.008221] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 245.731410] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 245.736811] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 245.744377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 246.242242] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 246.262910] Lustre: Skipped 1 previous similar message [ 246.265118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 246.267365] Lustre: Skipped 2 previous similar messages [ 248.276694] LustreError: 12199:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 248.339876] Lustre: server umount lustre-MDT0000 complete [ 250.772548] LustreError: 8402:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758733718 with bad export cookie 15895612092072264169 [ 250.781608] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 250.788938] LustreError: 8402:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 251.362796] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 251.369096] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 251.373901] Lustre: Skipped 2 previous similar messages [ 255.394711] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 255.398721] Lustre: Skipped 1 previous similar message [ 260.515476] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 260.515475] LustreError: 6503:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 260.515491] LustreError: 6503:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 5 previous similar messages [ 260.521641] Lustre: Skipped 1 previous similar message [ 265.184709] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 265.263346] LustreError: 12401:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 265.267419] LustreError: 12401:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 265.372591] Lustre: server umount lustre-MDT0001 complete [ 267.319394] LustreError: 12604:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 267.323344] LustreError: 12604:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 267.353561] Lustre: server umount lustre-OST0000 complete [ 269.367521] Lustre: server umount lustre-OST0001 complete [ 273.487599] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 279.508204] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 286.789542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 290.969640] hrtimer: interrupt took 3626044 ns [ 291.540035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 295.699435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 300.104442] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 305.049679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 305.102263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 305.260716] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 305.284325] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 305.334159] Lustre: lustre-MDT0000: new disk, initializing [ 305.403919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 305.426413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 308.671839] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 317.195951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 317.244469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 317.311731] Lustre: 14986:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 317.379449] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 317.386266] Lustre: Skipped 1 previous similar message [ 317.444276] Lustre: lustre-MDT0001: new disk, initializing [ 317.510161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 317.550300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 317.562271] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 320.312316] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 325.763925] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 328.443753] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 333.712922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 333.790531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 333.976854] Lustre: lustre-OST0000: new disk, initializing [ 333.979904] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 334.051024] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 335.407170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 335.422987] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 335.541085] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 338.366205] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 344.009648] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 346.766440] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 346.917961] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 349.365258] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 349.521393] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 351.203830] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 351.209075] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 351.217055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 351.221522] Lustre: Skipped 1 previous similar message [ 365.024137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 365.057132] LustreError: 17926:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 365.060143] LustreError: 17926:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 365.198855] Lustre: server umount lustre-OST0000 complete [ 368.610251] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 368.614077] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 368.618979] Lustre: Skipped 1 previous similar message [ 368.631502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 368.635714] Lustre: Skipped 4 previous similar messages [ 373.539299] LustreError: 18129:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 373.541823] LustreError: 18129:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 373.645309] Lustre: server umount lustre-MDT0000 complete [ 376.244841] LustreError: 14978:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758733843 with bad export cookie 15895612092072265702 [ 376.251081] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 376.463717] Lustre: server umount lustre-MDT0001 complete [ 380.815894] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 13:10:46 (1758733846) [ 394.691897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 395.101255] LustreError: 19467:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 395.155649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 397.832468] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 400.358100] LustreError: 19467:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 403.606699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 406.555314] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 410.220954] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 412.756867] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 414.283213] Lustre: Failing over lustre-MDT0000 [ 414.410144] LustreError: 20807:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 414.413453] LustreError: 20807:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 414.488213] Lustre: server umount lustre-MDT0000 complete [ 419.297804] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 419.298100] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 419.309543] Lustre: Skipped 1 previous similar message [ 419.311192] LustreError: 19462:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 419.324935] LustreError: 19462:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 424.417192] LustreError: 19466:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 424.426751] LustreError: 19466:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 430.935564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 431.025146] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 433.581127] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 436.193433] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 436.201860] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 436.216751] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 439.013781] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 446.434296] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 446.443172] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 446.445399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 446.447775] Lustre: Skipped 2 previous similar messages [ 446.450456] Lustre: Skipped 2 previous similar messages [ 448.212686] LustreError: 21788:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 448.215731] LustreError: 21788:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 448.258761] Lustre: server umount lustre-MDT0000 complete [ 448.485200] LustreError: 21603:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 448.492545] LustreError: 21603:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 450.811047] LustreError: 21789:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758733918 with bad export cookie 15895612092072267116 [ 450.811593] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 450.830211] LustreError: 21789:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 451.134590] Lustre: server umount lustre-MDT0001 complete [ 456.744702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 459.736464] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 465.383978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 465.613613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 465.616169] Lustre: Skipped 3 previous similar messages [ 468.133900] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 471.428745] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 474.036097] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 475.767659] Lustre: Failing over lustre-MDT0000 [ 475.976252] Lustre: server umount lustre-MDT0000 complete [ 476.834609] LustreError: 22460:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 476.854079] LustreError: 22460:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 492.199812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 492.266760] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 495.048064] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 497.635987] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 497.639565] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 497.644930] Lustre: Skipped 1 previous similar message [ 497.652243] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 500.265734] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 507.873709] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 507.879012] Lustre: Skipped 3 previous similar messages [ 507.882596] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 509.333494] LustreError: 24788:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 509.335784] LustreError: 24788:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 509.400076] Lustre: server umount lustre-MDT0000 complete [ 511.840653] LustreError: 22447:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758733979 with bad export cookie 15895612092072268047 [ 511.841145] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 511.850871] LustreError: 22447:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 1 previous similar message [ 512.139945] Lustre: server umount lustre-MDT0001 complete [ 518.104920] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 13:13:04 (1758733984) [ 527.952931] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 534.476789] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 539.905967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 543.482076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 547.277573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 551.328889] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 557.202084] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 564.131856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 564.176740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 564.336096] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 564.367666] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 564.424276] Lustre: lustre-MDT0000: new disk, initializing [ 564.498165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 566.932931] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 573.362630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 573.407111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 573.458587] Lustre: 28452:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 573.629702] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 575.976786] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 580.766932] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 582.780669] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 586.672085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 586.712217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 588.700180] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 588.730136] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 588.738924] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 589.560655] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 593.967688] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 598.326144] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 609.251756] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 609.253995] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 609.255080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 609.255086] Lustre: Skipped 4 previous similar messages [ 609.255319] LustreError: Skipped 1 previous similar message [ 609.274963] Lustre: Skipped 1 previous similar message [ 612.697489] LustreError: 31089:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 612.708236] LustreError: 31089:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 612.805627] Lustre: server umount lustre-OST0000 complete [ 621.204907] Lustre: server umount lustre-MDT0000 complete [ 623.443217] LustreError: 28446:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758734091 with bad export cookie 15895612092072268439 [ 623.446570] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 623.448206] LustreError: 28446:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 623.810198] Lustre: server umount lustre-MDT0001 complete [ 630.326934] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 632.007570] Key type lgssc unregistered [ 632.212907] LNet: 32023:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 632.218914] LNetError: 32023:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 632.229947] LNet: Removed LNI 192.168.203.142@tcp [ 632.742534] Key type .llcrypt unregistered [ 632.744588] Key type ._llcrypt unregistered [ 645.156666] Key type ._llcrypt registered [ 645.157931] Key type .llcrypt registered [ 645.219421] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 645.743375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 645.753525] alg: No test for adler32 (adler32-zlib) [ 646.706503] Lustre: Lustre: Build Version: 2.16.58_51_g2f926bd [ 646.833672] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 648.434614] Key type lgssc registered [ 649.109852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 653.492380] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 653.511875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 654.773419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 656.569637] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 661.379815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 661.596336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 663.815705] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 666.799899] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 668.860176] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 672.446639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 672.671201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 675.648411] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 678.505237] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 687.023970] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 696.191871] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 13:16:02 (1758734162) [ 697.202909] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 698.361113] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 702.575691] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 13:16:08 (1758734168) [ 708.180552] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 13:16:14 (1758734174) [ 708.820386] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 713.825192] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 713.829842] Lustre: Skipped 2 previous similar messages [ 715.077039] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 13:16:21 (1758734181) [ 719.955644] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 719.959560] Lustre: Skipped 4 previous similar messages [ 728.488143] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 13:16:34 (1758734194) [ 729.927089] Lustre: *** cfs_fail_loc=131b, val=2*** [ 731.915592] Lustre: *** cfs_fail_loc=131b, val=2*** [ 736.727298] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 13:16:42 (1758734202) [ 737.816484] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 737.819836] Lustre: Skipped 9 previous similar messages [ 746.214812] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 13:16:52 (1758734212) [ 756.708486] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 756.711046] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 756.716943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 758.754331] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 758.757632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 758.761503] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 758.774596] Lustre: Skipped 1 previous similar message [ 759.730117] LustreError: 39976:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 759.857749] Lustre: server umount lustre-OST0000 complete [ 763.873642] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 763.873773] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 763.874746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 763.887227] Lustre: Skipped 1 previous similar message [ 767.825541] LustreError: 40177:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 767.828401] LustreError: 40177:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 767.885273] Lustre: server umount lustre-MDT0000 complete [ 768.994348] LustreError: 33393:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 769.003050] LustreError: 33393:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 769.912692] LustreError: 38938:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758734237 with bad export cookie 15492178395934323504 [ 769.914913] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 769.921172] LustreError: 38938:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 770.037022] LustreError: 40379:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 770.039463] LustreError: 40379:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 770.140808] Lustre: server umount lustre-MDT0001 complete [ 775.407665] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 777.052755] Key type lgssc unregistered [ 777.227347] LNet: 40910:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 777.233129] LNetError: 40910:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 777.242553] LNet: Removed LNI 192.168.203.142@tcp [ 777.769147] Key type .llcrypt unregistered [ 777.772262] Key type ._llcrypt unregistered [ 787.213525] Key type ._llcrypt registered [ 787.214767] Key type .llcrypt registered [ 787.266955] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 796.162329] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 796.682925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 796.696366] alg: No test for adler32 (adler32-zlib) [ 797.664441] Lustre: Lustre: Build Version: 2.16.58_51_g2f926bd [ 797.806129] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 799.440848] Key type lgssc registered [ 800.204059] Lustre: Echo OBD driver; http://www.lustre.org/ [ 806.426502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 810.296468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 814.008162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 818.039864] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 821.868661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 821.903214] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 821.913303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 823.023419] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 823.045591] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 823.089148] Lustre: lustre-MDT0000: new disk, initializing [ 823.134342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 823.147564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 825.243542] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 831.783802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 831.827705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 831.873573] Lustre: 44014:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 831.912983] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 831.916489] Lustre: Skipped 1 previous similar message [ 831.975600] Lustre: lustre-MDT0001: new disk, initializing [ 832.054925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 832.097136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 832.104238] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 834.143140] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 838.603508] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 840.440275] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 843.868474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 843.904834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 844.038278] Lustre: lustre-OST0000: new disk, initializing [ 844.042138] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 844.083555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 847.078564] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 849.946205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 849.955684] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 850.033345] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 851.453154] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 853.651742] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 853.765439] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 856.036096] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 856.169641] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 860.134614] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 860.135295] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 860.141688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 860.148178] LustreError: Skipped 1 previous similar message [ 863.285779] LustreError: 46951:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 863.378614] Lustre: server umount lustre-OST0000 complete [ 867.809854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 867.813117] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 867.818301] Lustre: Skipped 1 previous similar message [ 867.820727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 867.823825] Lustre: Skipped 1 previous similar message [ 870.369790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 870.371275] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 871.492141] LustreError: 47152:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 871.494490] LustreError: 47152:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 871.543932] Lustre: server umount lustre-MDT0000 complete [ 873.347070] LustreError: 44796:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758734340 with bad export cookie 3748593186985602305 [ 873.351450] LustreError: 44796:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 1 previous similar message [ 873.351655] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 873.430573] LustreError: 47353:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 873.435186] LustreError: 47353:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 873.558777] Lustre: server umount lustre-MDT0001 complete [ 878.389634] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 13:19:04 (1758734344) [ 881.842630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 882.113628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 884.019593] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 887.265596] LustreError: 48072:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 887.273910] LustreError: 48072:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 888.401228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 888.594615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 890.404378] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 893.044243] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 894.798121] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 898.355484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 898.516098] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 901.180480] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 903.891217] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 911.253849] LustreError: 50271:0:(llog.c:430:llog_init_handle()) llog type is not specified! [ 911.769781] Lustre: Modifying parameter general.timeout=1 in log params [ 913.041566] Lustre: Modifying parameter general.timeout=4 in log params [ 913.044882] Lustre: Skipped 2 previous similar messages [ 915.155678] Lustre: Modifying parameter general.timeout=9 in log params [ 915.158451] Lustre: Skipped 4 previous similar messages [ 919.307155] Lustre: Modifying parameter general.timeout=20 in log params [ 919.309209] Lustre: Skipped 10 previous similar messages [ 927.656100] Lustre: Modifying parameter general.timeout=42 in log params [ 927.658391] Lustre: Skipped 21 previous similar messages [ 943.792377] Lustre: Modifying parameter general.timeout=84 in log params [ 943.794245] Lustre: Skipped 41 previous similar messages [ 967.962942] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 13:20:34 (1758734434) [ 969.283084] Lustre: 57176:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 970.977092] Lustre: 57372:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 970.993459] Lustre: 57372:0:(mgs_llog.c:1345:mgs_modify_param()) Skipped 1 previous similar message [ 973.548962] Lustre: 57666:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 973.554401] Lustre: 57666:0:(mgs_llog.c:1345:mgs_modify_param()) Skipped 2 previous similar messages [ 976.117731] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 in log params [ 976.122419] Lustre: Skipped 56 previous similar messages [ 985.343593] Lustre: 59007:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 985.356256] LustreError: 59007:0:(mgs_handler.c:1126:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 987.267097] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 13:20:53 (1758734453) [ 995.811137] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 995.813636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 995.817018] Lustre: Skipped 2 previous similar messages [ 995.821577] Lustre: Skipped 3 previous similar messages [ 998.277976] LustreError: 59704:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 998.281012] LustreError: 59704:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 998.319180] Lustre: server umount lustre-MDT0000 complete [ 999.804181] LustreError: 48052:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758734467 with bad export cookie 3748593186985603194 [ 999.806729] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 999.809173] LustreError: 48052:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 999.959170] Lustre: server umount lustre-MDT0001 complete [ 1011.634835] LustreError: 60106:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1011.638822] LustreError: 60106:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1011.736861] Lustre: server umount lustre-OST0000 complete [ 1015.546556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1017.218032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1018.773288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1020.401429] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1023.417983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1023.542856] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1023.677043] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1025.286960] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1028.911780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1028.968913] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1028.983199] Lustre: 61555:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1030.826064] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1033.006052] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1034.479271] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1037.255395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1037.345945] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1039.809943] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1041.973610] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1047.936478] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1047.941223] Lustre: Skipped 12 previous similar messages [ 1047.961000] Lustre: 63790:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1054.179381] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1054.179818] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1054.182678] LustreError: Skipped 1 previous similar message [ 1054.183711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1054.189530] Lustre: Skipped 1 previous similar message [ 1054.196143] Lustre: Skipped 1 previous similar message [ 1056.300449] LustreError: 64046:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1056.303697] LustreError: 64046:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1056.376664] Lustre: server umount lustre-OST0000 complete [ 1064.035206] Lustre: server umount lustre-MDT0000 complete [ 1064.417328] LustreError: 61565:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1064.425451] LustreError: 61565:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 1065.750455] LustreError: 61545:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758734533 with bad export cookie 3748593186985605658 [ 1065.755419] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1065.756651] LustreError: 61545:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1065.928286] Lustre: server umount lustre-MDT0001 complete [ 1070.118348] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 1071.374509] Key type lgssc unregistered [ 1071.519755] LNet: 64980:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1071.523154] LNetError: 64980:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1071.532367] LNet: Removed LNI 192.168.203.142@tcp [ 1071.879502] Key type .llcrypt unregistered [ 1071.880984] Key type ._llcrypt unregistered [ 1082.338588] Key type ._llcrypt registered [ 1082.339931] Key type .llcrypt registered [ 1082.381477] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1082.717154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1082.726165] alg: No test for adler32 (adler32-zlib) [ 1083.598303] Lustre: Lustre: Build Version: 2.16.58_51_g2f926bd [ 1083.695297] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 1085.296358] Key type lgssc registered [ 1085.768742] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1088.989432] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1089.000077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1090.270565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1092.038915] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1095.967793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1096.249459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1097.911437] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1100.168881] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1101.760285] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1104.702941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1104.951081] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1107.216861] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1109.365300] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1122.900743] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1126.881271] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1126.884469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1126.891412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1130.465881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1130.468030] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1131.902602] LustreError: 69078:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1131.929307] Lustre: server umount lustre-MDT0000 complete [ 1133.389316] LustreError: 66331:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758734600 with bad export cookie 11250207169989354790 [ 1133.392534] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1133.393552] LustreError: 66331:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1133.468751] LustreError: 69282:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1133.471454] LustreError: 69282:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1133.563906] Lustre: server umount lustre-MDT0001 complete [ 1145.269808] LustreError: 69482:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1145.272082] LustreError: 69482:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1145.351532] Lustre: server umount lustre-OST0000 complete [ 1148.806373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1150.242562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1151.752736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1153.307349] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1156.080127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1156.194438] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1156.211127] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1156.322745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1157.677668] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1160.704472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1160.742351] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1160.753524] Lustre: 70928:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1160.938869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1162.463457] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1164.450649] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1165.765043] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1168.449109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1168.527628] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1170.773463] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1172.730830] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1173.395405] Lustre: 73171:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1174.165637] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1177.062385] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1177.062648] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1177.068376] Lustre: Skipped 1 previous similar message [ 1177.071832] LustreError: Skipped 1 previous similar message [ 1177.074518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1177.077846] Lustre: Skipped 3 previous similar messages [ 1177.710862] LustreError: 73421:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1177.714393] LustreError: 73421:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1177.784858] Lustre: server umount lustre-OST0000 complete [ 1181.289201] Lustre: server umount lustre-MDT0000 complete [ 1181.665928] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1181.669690] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1181.674692] Lustre: Skipped 1 previous similar message [ 1181.679217] LustreError: 70940:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1181.688336] LustreError: 70940:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 1182.179208] LustreError: 72028:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1182.186370] LustreError: 72028:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 1182.773311] LustreError: 70920:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758734650 with bad export cookie 11250207169989355609 [ 1182.776616] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1182.778038] LustreError: 70920:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1182.849930] LustreError: 73828:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1182.852433] LustreError: 73828:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 1182.937594] Lustre: server umount lustre-MDT0001 complete [ 1187.115851] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 1188.324438] Key type lgssc unregistered [ 1188.457638] LNet: 74359:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1188.461330] LNetError: 74359:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1188.468453] LNet: Removed LNI 192.168.203.142@tcp [ 1188.810608] Key type .llcrypt unregistered [ 1188.811992] Key type ._llcrypt unregistered [ 1199.095768] Key type ._llcrypt registered [ 1199.096943] Key type .llcrypt registered [ 1199.138445] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1199.511279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1199.573076] alg: No test for adler32 (adler32-zlib) [ 1200.445897] Lustre: Lustre: Build Version: 2.16.58_51_g2f926bd [ 1200.547669] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 1202.144270] Key type lgssc registered [ 1202.579559] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1205.602109] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1205.612633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1206.849643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1208.440912] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1211.755699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1212.007438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1213.510422] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1215.675632] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1217.244387] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1220.119443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1220.370169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1222.979102] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1225.150369] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1238.245367] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1238.974574] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1238.977554] Lustre: Skipped 1 previous similar message [ 1240.059452] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1240.062099] Lustre: Skipped 2 previous similar messages [ 1242.215385] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1242.218262] Lustre: Skipped 5 previous similar messages [ 1246.550963] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1246.553611] Lustre: Skipped 10 previous similar messages [ 1254.823853] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1254.826379] Lustre: Skipped 21 previous similar messages [ 1271.051582] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1271.054696] Lustre: Skipped 42 previous similar messages [ 1279.006361] Lustre: 83488:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1280.033866] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 13:25:46 (1758734746) [ 1280.679918] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1281.362934] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 13:25:47 (1758734747) [ 1283.042799] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1283.045559] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1283.046083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1283.050480] Lustre: Skipped 1 previous similar message [ 1288.162624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1288.165401] Lustre: Skipped 2 previous similar messages [ 1288.559074] LustreError: 83782:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1288.626570] Lustre: server umount lustre-OST0000 complete [ 1293.283741] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1293.286718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1293.289069] Lustre: Skipped 1 previous similar message [ 1296.322555] LustreError: 83985:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1296.324982] LustreError: 83985:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1296.371907] Lustre: server umount lustre-MDT0000 complete [ 1298.035613] LustreError: 75712:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758734765 with bad export cookie 10291511226404136171 [ 1298.038383] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1298.040864] LustreError: 75712:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1298.109813] LustreError: 84187:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1298.111693] LustreError: 84187:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1298.214637] Lustre: server umount lustre-MDT0001 complete [ 1302.765785] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 1304.006750] Key type lgssc unregistered [ 1304.164469] LNet: 84717:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1304.168181] LNetError: 84717:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1304.174453] LNet: Removed LNI 192.168.203.142@tcp [ 1304.525847] Key type .llcrypt unregistered [ 1304.527067] Key type ._llcrypt unregistered [ 1306.115969] Key type ._llcrypt registered [ 1306.117395] Key type .llcrypt registered [ 1306.163368] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1307.757778] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules [ 1308.141934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1308.183657] alg: No test for adler32 (adler32-zlib) [ 1309.072156] Lustre: Lustre: Build Version: 2.16.58_51_g2f926bd [ 1309.167602] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 1310.760182] Key type lgssc registered [ 1311.228210] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1311.691210] LustreError: 85833:0:(lu_object.c:1390:lu_context_key_register()) cfs_race id 60d sleeping [ 1320.171423] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1323.801632] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1332.963667] LustreError: 85833:0:(lu_object.c:1390:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1332.974552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1334.218801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1335.751415] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1339.429824] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 13:26:45 (1758734805) [ 1341.339300] LustreError: 86872:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1341.343817] LustreError: 86872:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1341.348152] LustreError: 86872:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 1341.418195] LustreError: 87645:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1341.541027] Lustre: server umount lustre-MDT0000 complete [ 1347.057580] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 1348.323378] Key type lgssc unregistered [ 1348.480042] LNet: 88330:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1348.483689] LNetError: 88330:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1348.491498] LNet: Removed LNI 192.168.203.142@tcp [ 1348.820047] Key type .llcrypt unregistered [ 1348.821459] Key type ._llcrypt unregistered [ 1356.885172] Key type ._llcrypt registered [ 1356.886398] Key type .llcrypt registered [ 1356.932581] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1357.377684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1357.418347] alg: No test for adler32 (adler32-zlib) [ 1358.290834] Lustre: Lustre: Build Version: 2.16.58_51_g2f926bd [ 1358.389785] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 1359.976320] Key type lgssc registered [ 1360.430381] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1363.692097] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1363.707404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1364.962689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1366.541339] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1370.277823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1370.575517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1372.210571] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1374.514674] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1376.038494] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1378.534778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1378.765709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1380.938351] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1383.087224] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1417.602951] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 13:28:04 (1758734884) [ 1418.292464] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1419.007328] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 13:28:05 (1758734885) [ 1420.770361] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1420.771952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1420.775782] Lustre: Skipped 2 previous similar messages [ 1420.778505] Lustre: Skipped 1 previous similar message [ 1425.891103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1425.894386] Lustre: Skipped 3 previous similar messages [ 1426.437346] LustreError: 92432:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1426.471901] Lustre: server umount lustre-MDT0000 complete [ 1428.081439] LustreError: 90230:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758734895 with bad export cookie 16432547826683136601 [ 1428.085977] LustreError: 90230:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1428.086865] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1428.182221] LustreError: 92635:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1428.185109] LustreError: 92635:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1428.298255] Lustre: server umount lustre-MDT0001 complete [ 1440.180347] LustreError: 92835:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1440.183657] LustreError: 92835:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1440.382328] Lustre: server umount lustre-OST0000 complete [ 1445.508768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1445.716598] LustreError: 93464:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1445.727414] LustreError: 93464:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 1445.805770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1447.455487] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1450.977126] LustreError: 93464:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1451.208050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1451.614460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1453.358918] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1455.574399] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1457.063080] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1459.687675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1462.428262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1462.457444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1462.534644] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1462.538802] LustreError: 93452:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1462.549167] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1462.552763] LustreError: 95324:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -98 [ 1462.556260] LustreError: 95324:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 1462.558882] LustreError: 95324:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 1462.613768] Lustre: server umount lustre-OST0000 complete [ 1462.615654] LustreError: 95324:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1465.825802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1465.907254] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1465.911293] LustreError: 93452:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1465.915961] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1465.920580] LustreError: 95787:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -98 [ 1465.924163] LustreError: 95787:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 1465.926595] LustreError: 95787:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 1465.977789] Lustre: server umount lustre-OST0000 complete [ 1465.980165] LustreError: 95787:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1467.086517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1469.799093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1469.887931] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1469.890138] Lustre: Found index 0 for lustre-OST0000, updating log [ 1469.892861] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1469.926214] Lustre: lustre-OST0000: new disk, initializing [ 1469.928189] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1470.032843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1472.302251] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1473.796754] Lustre: Failing over lustre-OST0000 [ 1473.820231] LustreError: 96869:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1473.822633] LustreError: 96869:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1473.898654] Lustre: server umount lustre-OST0000 complete [ 1476.450698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1476.452644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1476.456195] Lustre: Skipped 1 previous similar message [ 1476.460730] Lustre: Skipped 1 previous similar message [ 1481.697921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1481.700612] Lustre: Skipped 1 previous similar message [ 1485.796379] LustreError: 97069:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1485.798689] LustreError: 97069:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1485.824809] Lustre: server umount lustre-MDT0000 complete [ 1486.817494] LustreError: 94193:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1486.823542] LustreError: 94193:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 1487.138552] LustreError: 93443:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758734954 with bad export cookie 16432547826683139331 [ 1487.142878] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1487.143803] LustreError: 93443:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1487.293912] Lustre: server umount lustre-MDT0001 complete [ 1490.524045] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 13:29:16 (1758734956) [ 1496.525205] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1500.389967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1500.568812] LustreError: 98637:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1500.601490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1502.087657] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1505.404381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1507.177328] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1508.304220] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1510.780960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1510.820912] LustreError: 100098:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1511.080598] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 1513.301377] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1515.514092] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1515.528482] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1515.529341] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 1515.531915] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 1516.671329] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1516.701062] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1516.730669] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1516.771272] Lustre: lustre-OST0001: new disk, initializing [ 1516.773616] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1519.136773] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1521.683770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1521.690494] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1521.728852] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1522.661676] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1526.533764] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1535.441434] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.42@tcp (stopping) [ 1536.481041] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 1536.483945] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1539.263911] LustreError: 101879:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 1539.266918] LustreError: 101879:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 1539.296518] Lustre: server umount lustre-MDT0001 complete [ 1540.354539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1540.551494] LustreError: 98664:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1542.904221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1542.949069] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1542.951636] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1542.969920] Lustre: 98624:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1543.273061] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1543.276246] Lustre: Skipped 3 previous similar messages [ 1544.553866] LustreError: 102600:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1544.557983] LustreError: 102600:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1544.562559] LustreError: 98628:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1544.595960] LustreError: 102614:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.203.142@tcp: cfg command failed: rc = -17 [ 1544.600415] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1544.600415] [ 1544.771862] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1548.436534] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 13:30:14 (1758735014) [ 1548.777405] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 1554.790638] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1561.823880] Lustre: 104484:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1570.678881] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1572.016989] Lustre: 105617:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1625.057082] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1625.060687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1625.065911] Lustre: Skipped 2 previous similar messages [ 1625.068090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1625.070637] Lustre: Skipped 3 previous similar messages [ 1630.153671] LustreError: 106053:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 1630.156483] LustreError: 106053:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1630.188465] Lustre: server umount lustre-MDT0000 complete [ 1630.688672] LustreError: 100872:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1630.694209] LustreError: 100872:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 6 previous similar messages [ 1631.564699] LustreError: 100913:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758735099 with bad export cookie 16432547826683140178 [ 1631.566054] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1631.568656] LustreError: 100913:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1631.730027] Lustre: server umount lustre-MDT0001 complete [ 1643.015985] Lustre: server umount lustre-OST0000 complete [ 1654.715813] Lustre: server umount lustre-OST0001 complete [ 1656.703128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1659.937924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1660.287920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1663.123425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1666.245297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1666.604479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1673.149862] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1677.088630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1677.098453] Lustre: lustre-MDT0000: reset Object Index mappings [ 1677.316180] LustreError: 109532:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1677.361323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1678.977711] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1682.568845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1682.577500] Lustre: lustre-MDT0001: reset Object Index mappings [ 1684.603639] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1685.867990] Lustre: 110649:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1688.691310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1691.507471] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1693.164038] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 1695.125612] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1696.755501] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:225) [ 1697.725765] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1701.868960] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 1701.869463] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:386 to 0x280000401:418) [ 1706.611815] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1708.143834] Lustre: 112500:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1733.394533] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 13:33:19 (1758735199) [ 1763.302902] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1763.304901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1763.307864] Lustre: Skipped 6 previous similar messages [ 1763.311617] Lustre: Skipped 6 previous similar messages [ 1768.072072] LustreError: 112848:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1768.074172] LustreError: 112848:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 1768.104740] Lustre: server umount lustre-MDT0000 complete [ 1768.420058] LustreError: 110274:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1768.426381] LustreError: 110274:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 5 previous similar messages [ 1769.314147] LustreError: 109511:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758735236 with bad export cookie 16432547826683187176 [ 1769.318111] LustreError: 109511:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1769.319891] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1773.537816] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1773.545020] Lustre: Skipped 1 previous similar message [ 1775.584555] Lustre: server umount lustre-MDT0001 complete [ 1783.107890] Lustre: server umount lustre-OST0000 complete [ 1790.714763] Lustre: server umount lustre-OST0001 complete [ 1793.071292] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 1796.817222] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1800.570528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1802.888815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1805.108977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1807.556985] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1809.985210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1810.007965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1810.098300] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1810.111110] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1810.148830] Lustre: lustre-MDT0000: new disk, initializing [ 1810.176791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1810.179403] Lustre: Skipped 3 previous similar messages [ 1810.186949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1811.587739] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1814.574502] Lustre: server umount lustre-MDT0000 complete [ 1815.539171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1817.961315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1817.990079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1818.123699] Lustre: Found index 0 for lustre-MDT0000, updating log [ 1818.130589] Lustre: 116719:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1818.137190] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 1819.726345] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1821.668186] Lustre: server umount lustre-MDT0000 complete [ 1824.835934] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 13:34:51 (1758735291) [ 1825.422877] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 1826.080195] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 13:34:52 (1758735292) [ 1832.575736] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1836.678757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1838.484650] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1842.035532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1842.067734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1842.110498] Lustre: 118737:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1842.139422] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1842.141831] Lustre: Skipped 1 previous similar message [ 1842.182938] Lustre: lustre-MDT0001: new disk, initializing [ 1842.240379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1842.244561] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1843.775100] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1846.185042] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1848.588170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1848.612987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1848.713381] Lustre: lustre-OST0000: new disk, initializing [ 1848.716150] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1849.405424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1849.411045] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1849.457114] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1851.027557] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1855.575066] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1855.602915] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1857.159609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1857.163960] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1857.211238] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1857.924951] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 1862.736524] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1864.100184] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2028.807818] Lustre: 127504:0:(osd_internal.h:1341:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 2028.811393] Lustre: 127504:0:(osd_handler.c:2079:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 2028.814581] Lustre: 127504:0:(osd_handler.c:2086:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 2028.818198] Lustre: 127504:0:(osd_handler.c:2096:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 2028.821687] Lustre: 127504:0:(osd_handler.c:2103:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 2028.824341] Lustre: 127504:0:(osd_handler.c:2110:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2128.335578] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 13:39:54 (1758735594) [ 2141.892053] LustreError: 127712:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2141.894623] LustreError: 127712:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 2141.964158] Lustre: server umount lustre-MDT0000 complete [ 2143.237399] LustreError: 120281:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758735610 with bad export cookie 16432547826683243792 [ 2143.241334] LustreError: 120281:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2143.242283] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2144.225400] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2144.226581] LustreError: 121924:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2144.226946] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2144.226952] Lustre: Skipped 4 previous similar messages [ 2144.230868] Lustre: Skipped 3 previous similar messages [ 2144.235521] LustreError: 121924:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 4 previous similar messages [ 2149.491984] Lustre: server umount lustre-MDT0001 complete [ 2165.216127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2165.255403] Lustre: server umount lustre-OST0000 complete [ 2172.737905] Lustre: server umount lustre-OST0001 complete [ 2175.077378] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 2178.596162] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 2182.348970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2184.924317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2187.286385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2189.636805] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2192.216038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2192.243273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2192.344899] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2192.361158] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2192.364094] Lustre: Skipped 1 previous similar message [ 2192.407944] Lustre: lustre-MDT0000: new disk, initializing [ 2192.409618] Lustre: Skipped 1 previous similar message [ 2192.441745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2192.444768] Lustre: Skipped 5 previous similar messages [ 2192.453188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2193.959863] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2198.741649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2198.780639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2198.826245] Lustre: 130496:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2198.955045] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2200.689634] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2204.442818] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2206.082542] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2209.024409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2209.056497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2209.176545] Lustre: lustre-OST0000: new disk, initializing [ 2209.178847] Lustre: Skipped 1 previous similar message [ 2209.181639] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2209.184296] Lustre: Skipped 2 previous similar messages [ 2211.206974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2211.210165] Lustre: Skipped 1 previous similar message [ 2211.212452] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2211.265124] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2211.724200] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2215.307944] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2219.301463] Lustre: lustre-MDD0000: changelog on [ 2220.393274] Lustre: lustre-MDD0001: changelog on [ 2664.553319] Lustre: lustre-MDD0001: changelog off [ 2665.182916] Lustre: 134883:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 2665.488821] Lustre: lustre-MDD0000: changelog off [ 2671.585170] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2671.587518] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2671.591316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2671.594305] Lustre: Skipped 3 previous similar messages [ 2672.097798] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2673.199792] LustreError: 139758:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2673.202933] LustreError: 139758:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 2673.274957] Lustre: server umount lustre-OST0000 complete [ 2675.169100] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2680.765159] Lustre: server umount lustre-MDT0000 complete [ 2682.036412] LustreError: 130489:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758736149 with bad export cookie 16432547826684043549 [ 2682.039804] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2682.041370] LustreError: 130489:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2682.193916] Lustre: server umount lustre-MDT0001 complete [ 2685.765022] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 2686.781096] Key type lgssc unregistered [ 2686.904534] LNet: 140691:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2686.908206] LNetError: 140691:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2686.919428] LNet: Removed LNI 192.168.203.142@tcp [ 2687.247163] Key type .llcrypt unregistered [ 2687.248149] Key type ._llcrypt unregistered [ 2693.374163] Key type ._llcrypt registered [ 2693.375703] Key type .llcrypt registered [ 2693.417348] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 2699.561727] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 2699.872529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2699.900025] alg: No test for adler32 (adler32-zlib) [ 2700.747913] Lustre: Lustre: Build Version: 2.16.58_51_g2f926bd [ 2700.820917] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 2702.392087] Key type lgssc registered [ 2702.732600] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2705.476528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2707.507832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2709.462377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2711.722575] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2714.039132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2714.064118] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2714.069849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2715.156595] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2715.171698] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2715.208494] Lustre: lustre-MDT0000: new disk, initializing [ 2715.234982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2715.242702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2716.576666] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2720.606204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2720.627210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2720.656788] Lustre: 143838:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2720.679531] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2720.682793] Lustre: Skipped 1 previous similar message [ 2720.712617] Lustre: lustre-MDT0001: new disk, initializing [ 2720.742425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2720.752973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2720.756096] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2722.001951] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2725.049864] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2726.181085] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2728.400029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2728.419714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2728.509448] Lustre: lustre-OST0000: new disk, initializing [ 2728.511273] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2728.554302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2730.413235] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2733.552860] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2734.056735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2734.059375] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2734.072153] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2741.189819] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_module obdecho/obdecho [ 2741.820770] LustreError: 146467:0:(echo_client.c:670:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 2741.823851] LustreError: 146467:0:(obd_config.c:783:class_setup()) setup ec failed (-22) [ 2742.420360] LustreError: 146563:0:(echo_client.c:1881:echo_md_handler()) Can not resolve the path : rc = -2 [ 2746.336798] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2746.339168] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2746.342980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2749.410038] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2749.410585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2749.414718] Lustre: Skipped 1 previous similar message [ 2749.416806] Lustre: Skipped 1 previous similar message [ 2751.675079] LustreError: 146903:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2751.706972] Lustre: server umount lustre-MDT0000 complete [ 2752.985320] LustreError: 143829:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758736220 with bad export cookie 3380969498148105870 [ 2752.988474] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2752.990693] LustreError: 143829:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2753.058129] LustreError: 147104:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2753.060645] LustreError: 147104:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2753.145738] Lustre: server umount lustre-MDT0001 complete [ 2764.659522] LustreError: 147304:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2764.661955] LustreError: 147304:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2764.731035] Lustre: server umount lustre-OST0000 complete [ 2771.598664] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 2774.718344] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 2777.885663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2780.060929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2782.171320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2784.352837] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2786.696341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2786.722871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2786.802567] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2786.814326] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2786.850608] Lustre: lustre-MDT0000: new disk, initializing [ 2786.874637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2786.882114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2788.265321] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2792.439144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2792.469494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2792.512967] Lustre: 150245:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2792.539858] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2792.542087] Lustre: Skipped 1 previous similar message [ 2792.579744] Lustre: lustre-MDT0001: new disk, initializing [ 2792.615361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2792.629729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2792.633980] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2794.063040] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2797.344116] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2798.648462] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2801.220230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2801.247038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2801.346782] Lustre: lustre-OST0000: new disk, initializing [ 2801.348986] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2801.382752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2803.480034] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2803.640249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2803.644021] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2803.692959] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2806.733284] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2808.251078] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2808.327906] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2809.751599] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2809.830684] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2813.922942] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2813.923063] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2813.924148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2813.925713] LustreError: Skipped 1 previous similar message [ 2813.930379] Lustre: Skipped 1 previous similar message [ 2816.558069] LustreError: 153185:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2816.559935] LustreError: 153185:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2816.625900] Lustre: server umount lustre-OST0000 complete [ 2818.528880] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2818.531432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2818.534886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2818.536633] Lustre: Skipped 1 previous similar message [ 2824.161247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2824.163928] Lustre: Skipped 3 previous similar messages [ 2832.352217] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2832.385766] LustreError: 153386:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2832.388210] LustreError: 153386:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2832.417911] Lustre: server umount lustre-MDT0000 complete [ 2833.637680] LustreError: 150237:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758736301 with bad export cookie 3380969498148106815 [ 2833.642496] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2833.796738] Lustre: server umount lustre-MDT0001 complete [ 2835.006995] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 13:51:41 (1758736301) [ 2835.580505] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 2836.208692] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 13:51:42 (1758736302) [ 2838.764291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2838.922614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2840.218471] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2843.313716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2844.776223] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2846.668718] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2848.034186] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2850.448167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2852.520695] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2854.420983] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2869.218148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2869.218872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2869.220066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2869.220072] Lustre: Skipped 2 previous similar messages [ 2869.222626] Lustre: Skipped 2 previous similar messages [ 2873.155839] LustreError: 156523:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2873.158622] LustreError: 156523:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2873.184057] Lustre: server umount lustre-MDT0000 complete [ 2874.337230] LustreError: 154258:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2874.342925] LustreError: 154258:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 4 previous similar messages [ 2874.420162] LustreError: 154240:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758736342 with bad export cookie 3380969498148107683 [ 2874.423786] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2874.424353] LustreError: 154240:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 2875.335286] LustreError: 154977:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2879.456871] LustreError: 154977:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2879.457170] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2879.467612] Lustre: Skipped 1 previous similar message [ 2880.654498] Lustre: server umount lustre-MDT0001 complete [ 2882.817576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2886.974990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2887.054121] LustreError: 157783:0:(llog.c:1602:llog_backup()) MGC192.168.203.142@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 2887.056323] Lustre: 157783:0:(mgc_request_server.c:726:mgc_llog_local_copy()) MGC192.168.203.142@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 2899.937084] LustreError: 157791:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2899.942222] LustreError: 157791:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 2905.030824] LustreError: 157791:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2915.270639] LustreError: 157791:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2915.275050] LustreError: 157791:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 7 previous similar messages [ 2931.654777] LustreError: 157808:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2931.660754] LustreError: 157808:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 14 previous similar messages [ 2947.680200] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 2947.681736] Lustre: 157783:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.142@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 2966.470593] LustreError: 157808:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2966.475537] LustreError: 157808:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 20 previous similar messages [ 2977.760290] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 2977.762174] Lustre: 154250:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.142@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 2988.520107] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2988.523204] Lustre: Skipped 2 previous similar messages [ 2989.779173] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2992.557434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2993.636230] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2995.680569] LustreError: 142054:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000079153b1c x1844168536196608/t0(0) o250->MGC192.168.203.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2997.133509] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 2998.976329] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2999.552996] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3000.807154] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3000.856120] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3003.461925] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3010.364760] LustreError: 159271:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3010.367638] LustreError: 159271:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 3010.395210] Lustre: server umount lustre-MDT0000 complete [ 3011.040853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3011.043103] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3011.046935] Lustre: Skipped 2 previous similar messages [ 3011.555102] LustreError: 154238:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758736479 with bad export cookie 3380969498148114137 [ 3011.558417] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3011.559051] LustreError: 154238:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3011.693608] Lustre: server umount lustre-MDT0001 complete [ 3023.219917] Lustre: server umount lustre-OST0000 complete [ 3029.788795] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 3032.779289] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 3036.080505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3038.165185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3040.050035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3041.984360] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3044.182395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3044.206529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3044.280522] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3044.290874] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3044.322958] Lustre: lustre-MDT0000: new disk, initializing [ 3044.348611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3045.649288] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3051.526124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3051.576268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3051.635634] Lustre: 162614:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3051.819486] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3054.325500] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3061.120858] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3063.648735] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3069.157762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3069.225485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3069.563623] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3069.571691] Lustre: Skipped 3 previous similar messages [ 3074.248239] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3078.712115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3078.722018] Lustre: Skipped 1 previous similar message [ 3078.735803] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3078.837905] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3080.514598] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3083.804087] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3084.114159] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3086.643517] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3086.848386] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3089.379974] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3089.382403] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3089.383645] LustreError: Skipped 1 previous similar message [ 3089.392609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3089.401570] Lustre: Skipped 7 previous similar messages [ 3102.177205] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3102.198094] LustreError: 165549:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3102.201386] LustreError: 165549:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3102.269934] Lustre: server umount lustre-OST0000 complete [ 3109.857544] LustreError: 162619:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3109.866239] LustreError: 162619:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 28 previous similar messages [ 3109.884706] Lustre: server umount lustre-MDT0000 complete [ 3111.408480] LustreError: 162605:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758736579 with bad export cookie 3380969498148130923 [ 3111.410921] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3111.414122] LustreError: 162605:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3111.590060] Lustre: server umount lustre-MDT0001 complete [ 3114.548505] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 13:56:21 (1758736581) [ 3116.838500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3118.559697] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3122.312379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3125.387121] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3130.148920] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3133.219132] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3139.779243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3144.890382] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3150.118935] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3159.896551] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 13:57:05 (1758736625) [ 3163.623623] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3163.632504] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3163.642236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3163.658490] Lustre: Skipped 4 previous similar messages [ 3163.668016] Lustre: Skipped 7 previous similar messages [ 3169.167139] LustreError: 169062:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3169.169944] LustreError: 169062:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3169.475088] Lustre: server umount lustre-OST0000 complete [ 3187.168115] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3187.263577] Lustre: server umount lustre-MDT0000 complete [ 3189.523807] LustreError: 166603:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758736657 with bad export cookie 3380969498148131805 [ 3189.530430] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3189.530615] LustreError: 166603:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3189.802397] Lustre: server umount lustre-MDT0001 complete [ 3196.304576] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 3197.901719] Key type lgssc unregistered [ 3198.072806] LNet: 170001:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3198.077393] LNetError: 170001:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3198.091341] LNet: Removed LNI 192.168.203.142@tcp [ 3198.507351] Key type .llcrypt unregistered [ 3198.508273] Key type ._llcrypt unregistered [ 3209.359534] Key type ._llcrypt registered [ 3209.361537] Key type .llcrypt registered [ 3209.420229] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 3209.962136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3209.981650] alg: No test for adler32 (adler32-zlib) [ 3210.946780] Lustre: Lustre: Build Version: 2.16.58_51_g2f926bd [ 3211.120965] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 3212.768160] Key type lgssc registered [ 3213.446624] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3218.611864] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3218.638982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3235.235983] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3241.376196] LustreError: 171202:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.142@tcp: failed processing log, type 4: rc = -110 [ 3247.520858] LustreError: 171202:0:(llog_osd.c:243:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 3247.536619] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 3247.541696] LustreError: MGC192.168.203.142@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 3247.547125] LustreError: 171202:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-OST0000: -22 [ 3247.551252] LustreError: 171202:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -22 [ 3247.555320] LustreError: 171202:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 3247.560271] LustreError: 171202:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 3247.645414] Lustre: server umount lustre-OST0000 complete [ 3247.647045] LustreError: 171202:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 3253.566672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3254.863305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3257.571216] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3262.509585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3262.726526] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3265.020451] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3268.320238] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3270.507979] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3274.209749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3274.316949] LustreError: 173291:0:(llog_osd.c:243:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 3274.321930] Lustre: 173291:0:(mgc_request_server.c:706:mgc_llog_local_copy()) MGC192.168.203.142@tcp: invalid local config log lustre-OST0000: rc = -22 [ 3274.382411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3277.045427] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3279.384281] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3283.425155] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3283.428122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3283.433606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3284.961985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3284.962979] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3284.964440] Lustre: Skipped 1 previous similar message [ 3284.969974] Lustre: Skipped 1 previous similar message [ 3286.396940] LustreError: 173943:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3286.398670] LustreError: 173943:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3286.428194] Lustre: server umount lustre-MDT0000 complete [ 3287.768225] LustreError: 173320:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758736755 with bad export cookie 274369480263547468 [ 3287.771311] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3287.773655] LustreError: 173320:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3287.842186] LustreError: 174146:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3287.845333] LustreError: 174146:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3287.944251] Lustre: server umount lustre-MDT0001 complete [ 3299.890187] LustreError: 174346:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3299.892134] LustreError: 174346:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3299.972015] Lustre: server umount lustre-OST0000 complete [ 3303.655887] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 3307.607160] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 3311.695083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3314.789839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3317.496943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3320.410031] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3323.254664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3323.288841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3323.389754] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3323.409149] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3323.450142] Lustre: lustre-MDT0000: new disk, initializing [ 3323.486261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3323.495778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3325.121479] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3330.024605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3330.049577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3330.093442] Lustre: 176685:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3330.125983] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3330.128951] Lustre: Skipped 1 previous similar message [ 3330.167929] Lustre: lustre-MDT0001: new disk, initializing [ 3330.209316] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3330.224802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3330.229057] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3331.804943] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3335.139101] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3336.435833] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3338.833492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3338.865675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3338.963779] Lustre: lustre-OST0000: new disk, initializing [ 3338.966426] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3338.998207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3341.015926] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3342.857270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3342.860782] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3342.912590] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3344.493728] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3346.181546] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3346.278366] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3347.751429] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3347.842808] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3353.058961] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3353.059080] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3353.061253] LustreError: Skipped 1 previous similar message [ 3353.066699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3353.070570] Lustre: Skipped 2 previous similar messages [ 3354.613107] LustreError: 179623:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3354.615413] LustreError: 179623:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3354.687180] Lustre: server umount lustre-OST0000 complete [ 3356.128896] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3356.133225] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3356.136865] Lustre: Skipped 1 previous similar message [ 3356.139598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3363.298527] LustreError: 177481:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3363.305199] LustreError: 177481:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 3364.258060] LustreError: 179825:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 3364.262372] LustreError: 179825:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3364.287979] Lustre: server umount lustre-MDT0000 complete [ 3365.857598] LustreError: 176676:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758736833 with bad export cookie 274369480263548182 [ 3365.860559] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3365.865780] LustreError: 176676:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3366.061519] Lustre: server umount lustre-MDT0001 complete [ 3369.988294] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 14:00:36 (1758736836) [ 3377.225980] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 3378.465873] Key type lgssc unregistered [ 3378.615547] LNet: 181209:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3378.618974] LNetError: 181209:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3378.626503] LNet: Removed LNI 192.168.203.142@tcp [ 3378.985482] Key type .llcrypt unregistered [ 3378.986840] Key type ._llcrypt unregistered [ 3386.975714] Key type ._llcrypt registered [ 3386.976977] Key type .llcrypt registered [ 3387.024166] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 3387.409778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3387.460173] alg: No test for adler32 (adler32-zlib) [ 3388.332499] Lustre: Lustre: Build Version: 2.16.58_51_g2f926bd [ 3388.424178] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 3390.008142] Key type lgssc registered [ 3390.442060] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3393.666612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3393.679252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3394.908359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3396.455237] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3399.745083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3399.928851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3401.606937] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3403.928352] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3405.539500] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3408.507418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3408.591520] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3408.593509] LustreError: 184036:0:(llog.c:1642:llog_backup()) MGC192.168.203.142@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3408.598198] Lustre: 184036:0:(mgc_request_server.c:703:mgc_llog_local_copy()) MGC192.168.203.142@tcp: can't backup local config lustre-OST0000: rc = -28 [ 3408.640856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3410.834661] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3412.932541] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3413.743977] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3413.745743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3413.747921] Lustre: Skipped 2 previous similar messages [ 3413.750325] Lustre: Skipped 1 previous similar message [ 3419.106308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3419.108855] Lustre: Skipped 3 previous similar messages [ 3424.226486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3424.228929] Lustre: Skipped 2 previous similar messages [ 3428.320156] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3428.351933] LustreError: 184688:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3428.380139] Lustre: server umount lustre-MDT0000 complete [ 3429.345418] LustreError: 182397:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3429.351967] LustreError: 182397:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 4 previous similar messages [ 3429.794470] LustreError: 182364:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758736897 with bad export cookie 10850248191991872199 [ 3429.798060] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3429.800049] LustreError: 182364:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3429.872831] LustreError: 184891:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3429.875691] LustreError: 184891:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3429.965854] Lustre: server umount lustre-MDT0001 complete [ 3441.657936] LustreError: 185092:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3441.659760] LustreError: 185092:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3441.735166] Lustre: server umount lustre-OST0000 complete [ 3445.092951] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 3448.555169] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 3452.135377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3454.612450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3456.832251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3459.162118] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3461.662198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3461.689953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3461.775190] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3461.785292] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3461.823477] Lustre: lustre-MDT0000: new disk, initializing [ 3461.846850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3461.854628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3463.339175] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3467.691327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3467.722843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3467.766388] Lustre: 187425:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3467.791571] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3467.793391] Lustre: Skipped 1 previous similar message [ 3467.795364] LustreError: 188190:0:(llog.c:1642:llog_backup()) MGC192.168.203.142@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 3467.799488] LustreError: 188190:0:(llog.c:1642:llog_backup()) Skipped 1 previous similar message [ 3467.802791] Lustre: 188190:0:(mgc_request_server.c:726:mgc_llog_local_copy()) MGC192.168.203.142@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 3467.815274] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3467.818165] Lustre: Skipped 1 previous similar message [ 3467.858854] Lustre: lustre-MDT0001: new disk, initializing [ 3467.894212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3467.907750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3467.911485] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3469.274706] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3472.493403] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3473.684355] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3475.910124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3475.941880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3476.031428] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3476.033085] Lustre: Skipped 1 previous similar message [ 3476.035132] LustreError: 189144:0:(llog.c:1642:llog_backup()) MGC192.168.203.142@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3476.040028] LustreError: 189144:0:(llog.c:1642:llog_backup()) Skipped 1 previous similar message [ 3476.043129] Lustre: 189144:0:(mgc_request_server.c:726:mgc_llog_local_copy()) MGC192.168.203.142@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 3476.048702] Lustre: 189144:0:(mgc_request_server.c:726:mgc_llog_local_copy()) Skipped 1 previous similar message [ 3476.060498] Lustre: lustre-OST0000: new disk, initializing [ 3476.063136] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3476.093727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3478.034767] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3479.224756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3479.228185] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3479.256227] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3481.176760] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3482.449968] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3482.519796] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3483.837485] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3483.914849] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3484.642612] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3484.643029] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3484.644605] LustreError: Skipped 1 previous similar message [ 3484.645142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3484.649629] Lustre: Skipped 1 previous similar message [ 3486.638258] LustreError: 190362:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3486.640013] LustreError: 190362:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3486.699339] Lustre: server umount lustre-OST0000 complete [ 3488.736969] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3488.740086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3488.745964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3488.748822] Lustre: Skipped 1 previous similar message [ 3494.077091] LustreError: 190562:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3494.079231] LustreError: 190562:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3494.108893] Lustre: server umount lustre-MDT0000 complete [ 3494.881045] LustreError: 187435:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3494.886337] LustreError: 187435:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 3495.387157] LustreError: 187415:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758736962 with bad export cookie 10850248191991872899 [ 3495.389766] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3495.391549] LustreError: 187415:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3495.532614] Lustre: server umount lustre-MDT0001 complete [ 3498.682254] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 14:02:45 (1758736965) [ 3504.487755] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 3508.134281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3508.324796] LustreError: 192130:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3508.347153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3509.818798] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3512.883126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3514.403347] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3515.530532] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3517.848277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3520.545705] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3525.879754] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3525.944639] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3526.087263] Lustre: lustre-OST0001: new disk, initializing [ 3526.094248] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3530.698955] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3530.874170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3530.881660] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3530.944247] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3539.558672] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3550.405344] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3557.262601] Lustre: DEBUG MARKER: ADD OST3 [ 3562.469720] loop: module loaded [ 3562.563394] loop0: detected capacity change from 0 to 204800000 [ 3562.715961] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3562.721947] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3562.737906] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3562.945787] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3568.638506] loop0: detected capacity change from 0 to 204800000 [ 3569.435412] Lustre: DEBUG MARKER: STOP OST3 [ 3572.762458] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3572.813197] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3572.923480] Lustre: lustre-OST0003: new disk, initializing [ 3572.928169] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 3573.016424] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 3573.019349] Lustre: Skipped 3 previous similar messages [ 3574.125804] Lustre: *** cfs_fail_loc=2109, val=2*** [ 3574.189289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 3574.193657] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 3574.222289] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 3575.119762] Lustre: Failing over lustre-OST0003 [ 3575.163090] LustreError: 196763:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 3575.173199] LustreError: 196763:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3575.208051] Lustre: server umount lustre-OST0003 complete [ 3576.105937] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3576.160368] LustreError: 196590:0:(osp_precreate.c:1280:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 3579.374231] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3579.374292] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3579.381495] Lustre: Skipped 1 previous similar message [ 3579.386996] LustreError: 193584:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3579.398962] LustreError: 193584:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 3581.760959] Lustre: DEBUG MARKER: START OST3 again [ 3584.489860] LustreError: 194744:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3584.498375] LustreError: 194744:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 3585.489142] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3585.641155] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 3586.739336] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3587.062498] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 3587.063450] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3588.686781] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3604.779953] Lustre: Failing over lustre-OST0003 [ 3604.815101] LustreError: 198724:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 3604.820860] LustreError: 198724:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3604.845803] Lustre: server umount lustre-OST0003 complete [ 3605.974265] LustreError: 193584:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3605.981191] LustreError: 193584:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 3606.501618] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3606.507479] Lustre: Skipped 1 previous similar message [ 3620.833942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3620.837384] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3620.843671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3620.847317] Lustre: Skipped 2 previous similar messages [ 3624.561832] Lustre: server umount lustre-MDT0000 complete [ 3626.507649] LustreError: 192109:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758737094 with bad export cookie 10850248191991873767 [ 3626.509486] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3626.513169] LustreError: 192109:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3626.979353] LustreError: 193602:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3626.987681] LustreError: 193602:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 8 previous similar messages [ 3632.863328] Lustre: server umount lustre-MDT0001 complete [ 3635.191167] Lustre: server umount lustre-OST0000 complete [ 3637.334562] LustreError: 199603:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3637.337406] LustreError: 199603:0:(obd_class.h:479:obd_check_dev()) Skipped 27 previous similar messages [ 3637.443755] Lustre: server umount lustre-OST0001 complete [ 3640.678798] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 3644.718959] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 3649.287753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3652.618085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3655.703333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3658.624626] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3662.154713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3662.207405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3662.334420] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3662.352963] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3662.404797] Lustre: lustre-MDT0000: new disk, initializing [ 3662.456142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3662.458672] Lustre: Skipped 1 previous similar message [ 3662.467163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3664.598263] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3669.641207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3669.668696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3669.706937] Lustre: 201786:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3669.826786] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3671.485370] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3675.813408] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3678.307494] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3682.204037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3682.258932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3682.413341] Lustre: lustre-OST0000: new disk, initializing [ 3682.416251] Lustre: Skipped 1 previous similar message [ 3682.419407] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3682.424422] Lustre: Skipped 2 previous similar messages [ 3683.665703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3683.669408] Lustre: Skipped 1 previous similar message [ 3683.671770] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3683.751507] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3685.783530] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3689.927233] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3691.761639] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3691.868198] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3693.738769] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3693.839085] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3699.173404] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3699.173992] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3699.176710] LustreError: Skipped 1 previous similar message [ 3699.183248] Lustre: Skipped 5 previous similar messages [ 3699.186396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3699.192940] Lustre: Skipped 9 previous similar messages [ 3700.915589] Lustre: server umount lustre-OST0000 complete [ 3709.482047] LustreError: 204924:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3709.486272] LustreError: 204924:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 3709.558230] Lustre: server umount lustre-MDT0000 complete [ 3712.101272] LustreError: 201777:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758737179 with bad export cookie 10850248191991876049 [ 3712.104616] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3712.118894] LustreError: 201777:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3712.402406] Lustre: server umount lustre-MDT0001 complete [ 3714.839994] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 14:06:20 (1758737180) [ 3725.111473] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 3731.665509] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 3738.396270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3742.893515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3747.129211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3751.573204] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3756.111879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3756.153250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3756.308498] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3756.336549] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3756.397572] Lustre: lustre-MDT0000: new disk, initializing [ 3756.480452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3758.818380] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3765.644169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3765.678804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3765.735801] Lustre: 207968:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3765.906590] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3768.135774] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3772.975244] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3775.210071] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3779.034465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3779.074831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3780.810706] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3782.136175] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3786.497527] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3788.674965] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3788.785455] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3790.749198] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3790.856181] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3796.452178] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3796.452342] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3796.457704] LustreError: Skipped 1 previous similar message [ 3796.459586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3796.470096] Lustre: Skipped 4 previous similar messages [ 3796.475979] Lustre: Skipped 4 previous similar messages [ 3806.176128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3806.298897] Lustre: server umount lustre-OST0000 complete [ 3814.667123] Lustre: server umount lustre-MDT0000 complete [ 3816.345783] LustreError: 208748:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758737283 with bad export cookie 10850248191991876931 [ 3816.349469] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3816.350094] LustreError: 208748:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3816.521943] Lustre: server umount lustre-MDT0001 complete [ 3820.473409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3820.786896] LustreError: 211784:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3820.800272] LustreError: 211784:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 3820.836945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3820.839434] Lustre: Skipped 5 previous similar messages [ 3822.659577] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3826.830138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3828.835190] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3831.778218] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3834.288351] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3840.651615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3846.943155] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 3852.836840] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3863.009860] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3863.016504] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3863.022936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3863.022945] Lustre: Skipped 5 previous similar messages [ 3863.038129] Lustre: Skipped 3 previous similar messages [ 3869.002757] LustreError: 214126:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3869.017515] LustreError: 214126:0:(obd_class.h:479:obd_check_dev()) Skipped 31 previous similar messages [ 3869.330348] Lustre: server umount lustre-OST0000 complete [ 3888.096132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3888.275843] Lustre: server umount lustre-MDT0000 complete [ 3888.637303] LustreError: 212897:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3888.674703] LustreError: 212897:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 3892.206840] LustreError: 211764:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758737359 with bad export cookie 10850248191991877813 [ 3892.220352] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3892.232052] LustreError: 211764:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3892.820856] Lustre: server umount lustre-MDT0001 complete [ 3989.404309] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 3991.547233] Key type lgssc unregistered [ 3991.797174] LNet: 215108:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3991.802206] LNetError: 215108:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3991.812894] LNet: Removed LNI 192.168.203.142@tcp [ 3992.574169] Key type .llcrypt unregistered [ 3992.577662] Key type ._llcrypt unregistered [ 4009.613961] Key type ._llcrypt registered [ 4009.616465] Key type .llcrypt registered [ 4009.746321] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 4021.109130] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 4021.702276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4021.746379] alg: No test for adler32 (adler32-zlib) [ 4022.788345] Lustre: Lustre: Build Version: 2.16.58_51_g2f926bd [ 4022.997965] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 4024.673189] Key type lgssc registered [ 4025.590348] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4032.638509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4037.887243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4042.412781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4047.754864] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4053.537303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4053.593748] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4053.613628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4054.825031] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4054.870045] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4054.923309] Lustre: lustre-MDT0000: new disk, initializing [ 4054.999868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4055.022967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4058.135863] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4065.740382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4065.789526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4065.845174] Lustre: 218414:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4065.898965] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4065.903306] Lustre: Skipped 1 previous similar message [ 4065.949566] Lustre: lustre-MDT0001: new disk, initializing [ 4066.064638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4066.094864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4066.104086] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4068.770703] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4074.525780] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4077.786644] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4083.573460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4083.632552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4083.814751] Lustre: lustre-OST0000: new disk, initializing [ 4083.826813] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4083.935903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4089.341503] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4093.015543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4093.028699] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4093.153818] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4096.112626] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4099.834324] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4100.036817] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4103.315863] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4103.453594] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4108.769265] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4108.773895] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4108.784295] LustreError: Skipped 1 previous similar message [ 4108.794567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4110.982264] LustreError: 221355:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4111.129362] Lustre: server umount lustre-OST0000 complete [ 4113.891020] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4113.891667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4113.899845] Lustre: Skipped 2 previous similar messages [ 4113.901916] Lustre: Skipped 1 previous similar message [ 4119.012255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4119.015143] Lustre: Skipped 1 previous similar message [ 4124.135904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4124.138324] Lustre: Skipped 2 previous similar messages [ 4128.224131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4128.320220] LustreError: 221556:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4128.329406] LustreError: 221556:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4128.418202] Lustre: server umount lustre-MDT0000 complete [ 4131.017972] LustreError: 218406:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758737598 with bad export cookie 4754812899274810626 [ 4131.025408] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4131.110949] LustreError: 221759:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4131.114081] LustreError: 221759:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4131.261285] Lustre: server umount lustre-MDT0001 complete [ 4142.138591] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 4148.638235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4148.865719] LustreError: 222932:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4148.876449] LustreError: 222932:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 4148.919736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4151.249813] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4154.337396] LustreError: 222932:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4156.807384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4157.029199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4159.353482] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4161.276034] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4165.036221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4165.232964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4168.418449] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4173.853284] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4173.910601] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4174.006436] Lustre: lustre-OST0001: new disk, initializing [ 4174.010897] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4177.686181] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4180.532693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4180.546829] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4180.627524] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4185.879863] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4190.625808] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4202.850405] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 14:14:28 (1758737668) [ 4204.185932] LustreError: 226310:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4204.191341] LustreError: 226310:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4204.234979] Lustre: server umount lustre-MDT0000 complete [ 4206.388659] LustreError: 222911:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758737673 with bad export cookie 4754812899274811501 [ 4206.394063] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4206.397575] LustreError: 222911:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4206.562594] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4206.566668] LustreError: 222927:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4206.567935] Lustre: Skipped 2 previous similar messages [ 4206.573285] LustreError: 222927:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 4206.671866] Lustre: server umount lustre-MDT0001 complete [ 4210.751155] LustreError: 226711:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4210.761607] LustreError: 226711:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 4210.800701] Lustre: server umount lustre-OST0000 complete [ 4213.718066] Lustre: server umount lustre-OST0001 complete [ 4219.145060] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 4226.775656] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 4235.519308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4240.986983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4244.808843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4248.519150] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4252.277869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4252.314690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4252.431908] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4252.448609] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4252.495637] Lustre: lustre-MDT0000: new disk, initializing [ 4252.537118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4252.540142] Lustre: Skipped 1 previous similar message [ 4252.549311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4254.666597] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4262.106261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4262.157623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4262.215380] Lustre: 229097:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4262.252091] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4262.254715] Lustre: Skipped 1 previous similar message [ 4262.313330] Lustre: lustre-MDT0001: new disk, initializing [ 4262.392112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4262.402761] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4265.088629] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4270.242778] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4272.442266] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4277.127691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4277.172724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4279.262787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4279.267919] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4279.368192] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4280.958468] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4285.468891] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4287.591917] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4287.705563] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4289.604412] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4289.718443] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4294.627903] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4294.629969] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4294.636863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4294.643620] Lustre: Skipped 1 previous similar message [ 4294.651045] Lustre: Skipped 1 previous similar message [ 4304.864115] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4304.867319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4304.871974] Lustre: Skipped 3 previous similar messages [ 4304.904213] LustreError: 232035:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4304.910799] LustreError: 232035:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4305.037737] Lustre: server umount lustre-OST0000 complete [ 4308.450518] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4308.453699] LustreError: Skipped 1 previous similar message [ 4308.455045] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4313.394338] Lustre: server umount lustre-MDT0000 complete [ 4315.105969] LustreError: 229109:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4315.112102] LustreError: 229109:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 4315.468973] LustreError: 229089:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758737783 with bad export cookie 4754812899274813636 [ 4315.471527] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4315.475272] LustreError: 229089:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4315.774811] Lustre: server umount lustre-MDT0001 complete [ 4320.284304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4320.526904] LustreError: 232917:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4320.569820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4320.572466] Lustre: Skipped 2 previous similar messages [ 4322.759806] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4327.374818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4329.907414] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4333.199167] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4335.327888] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4339.121387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4342.434100] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4345.415317] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4489.188706] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4489.194457] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4489.205080] Lustre: Skipped 1 previous similar message [ 4489.208688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4489.212561] Lustre: Skipped 3 previous similar messages [ 4495.161246] LustreError: 235200:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4495.184340] LustreError: 235200:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4495.296979] Lustre: server umount lustre-OST0000 complete [ 4503.654528] Lustre: server umount lustre-MDT0000 complete [ 4506.084273] LustreError: 232897:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758737973 with bad export cookie 4754812899274814518 [ 4506.086761] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4506.090210] LustreError: 232897:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4506.329243] Lustre: server umount lustre-MDT0001 complete [ 4610.871469] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 4612.696720] Key type lgssc unregistered [ 4612.992972] LNet: 236169:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4613.008199] LNetError: 236169:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4614.054778] LNet: Removed LNI 192.168.203.142@tcp [ 4614.592617] Key type .llcrypt unregistered [ 4614.595108] Key type ._llcrypt unregistered [ 4628.000757] Key type ._llcrypt registered [ 4628.002315] Key type .llcrypt registered [ 4628.080129] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 4636.176343] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 4636.652516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4636.667486] alg: No test for adler32 (adler32-zlib) [ 4637.549461] Lustre: Lustre: Build Version: 2.16.58_51_g2f926bd [ 4637.647992] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 4639.248149] Key type lgssc registered [ 4639.796149] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4644.132834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4647.295037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4650.060357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4653.304846] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4661.358066] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 4667.401441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4667.452550] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4667.466253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4668.638770] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4668.669677] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4668.735589] Lustre: lustre-MDT0000: new disk, initializing [ 4668.791892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4668.808133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4671.594670] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4678.635407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4678.680402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4678.734802] Lustre: 240174:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4678.793855] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4678.797595] Lustre: Skipped 1 previous similar message [ 4678.862549] Lustre: lustre-MDT0001: new disk, initializing [ 4678.955147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4678.980906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4678.990103] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4681.533788] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4684.736572] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4688.745547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4688.796368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4689.023773] Lustre: lustre-OST0000: new disk, initializing [ 4689.029542] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4689.102279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4692.136900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4692.142212] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4692.257714] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4693.351784] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4700.942905] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4700.986649] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4701.107843] Lustre: lustre-OST0001: new disk, initializing [ 4701.112181] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4701.185470] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4705.280735] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4707.402943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4707.412385] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4707.515851] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4715.085733] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4722.314181] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4733.413458] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4733.414658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4733.421321] Lustre: Skipped 3 previous similar messages [ 4733.425866] Lustre: Skipped 3 previous similar messages [ 4738.531508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4738.543552] Lustre: Skipped 3 previous similar messages [ 4739.376184] LustreError: 243697:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4739.510053] Lustre: server umount lustre-MDT0000 complete [ 4741.909591] LustreError: 240166:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758738209 with bad export cookie 10254942860373453101 [ 4741.910837] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4741.915196] LustreError: 240166:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4742.007816] LustreError: 243899:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 4742.009938] LustreError: 243899:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4742.140092] Lustre: server umount lustre-MDT0001 complete [ 4754.367120] LustreError: 244099:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4754.373621] LustreError: 244099:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4754.398367] Lustre: server umount lustre-OST0000 complete [ 4765.439294] LustreError: 244301:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4765.443647] LustreError: 244301:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4765.560701] Lustre: server umount lustre-OST0001 complete [ 4773.365829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4778.253257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4778.806548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4783.678311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4788.838147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4789.442030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4801.058190] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 4806.964720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4806.976907] Lustre: lustre-MDT0000: reset Object Index mappings [ 4807.309556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4809.951289] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4812.772994] LustreError: 247611:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4812.792616] LustreError: 247611:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 4815.260305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4815.273328] Lustre: lustre-MDT0001: reset Object Index mappings [ 4815.525684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4817.299689] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4818.678455] Lustre: 248721:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4822.057020] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4824.917298] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4826.342673] LustreError: 249077:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4829.007745] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4831.783030] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4831.786473] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4831.787573] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 4836.304411] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4838.613438] Lustre: 250566:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4846.561376] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4846.565108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4846.570838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4851.734669] LustreError: 250668:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4851.738199] LustreError: 250668:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4851.800385] Lustre: server umount lustre-MDT0000 complete [ 4852.195350] LustreError: 250671:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4852.208197] LustreError: 250671:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 4854.048146] LustreError: 247590:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758738321 with bad export cookie 10254942860373459891 [ 4854.052793] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4854.053414] LustreError: 247590:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4854.306219] Lustre: server umount lustre-MDT0001 complete [ 4866.570891] LustreError: 251072:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 4866.577696] LustreError: 251072:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 4866.616072] Lustre: server umount lustre-OST0000 complete [ 4879.398965] Lustre: server umount lustre-OST0001 complete [ 4891.349136] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 14:25:57 (1758738357) [ 4901.209478] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 4906.687360] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 4911.677742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4914.922615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4917.626851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4921.047164] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4924.949891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4924.993155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4925.114786] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4925.133322] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4925.178633] Lustre: lustre-MDT0000: new disk, initializing [ 4925.233697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4925.236048] Lustre: Skipped 2 previous similar messages [ 4925.247640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4927.215754] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4933.633558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4933.669957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4933.724972] Lustre: 254792:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4933.760787] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4933.764802] Lustre: Skipped 1 previous similar message [ 4933.821342] Lustre: lustre-MDT0001: new disk, initializing [ 4933.921120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4933.926447] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4936.124986] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4940.670980] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4942.399269] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4945.812063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4945.847502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4947.089706] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4947.152395] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4948.869546] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4953.133346] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4955.323214] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4955.520446] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4958.486801] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4958.634224] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4962.788076] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4962.788211] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4962.793470] Lustre: Skipped 4 previous similar messages [ 4962.794179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4962.802174] Lustre: Skipped 4 previous similar messages [ 4965.946388] LustreError: 257733:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4965.952050] LustreError: 257733:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4966.055829] Lustre: server umount lustre-OST0000 complete [ 4969.953530] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4969.956883] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4969.966165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4974.414284] Lustre: server umount lustre-MDT0000 complete [ 4976.317587] LustreError: 255575:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758738443 with bad export cookie 10254942860373461984 [ 4976.321703] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4976.324494] LustreError: 255575:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4976.592048] Lustre: server umount lustre-MDT0001 complete [ 4987.012178] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 4994.282729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4994.566175] LustreError: 259310:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4994.606389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4994.609646] Lustre: Skipped 2 previous similar messages [ 4997.904135] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 4999.651679] LustreError: 259310:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5003.303162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5006.505105] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5008.811446] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5013.570900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5017.681693] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5023.460233] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5023.517198] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5023.631866] Lustre: lustre-OST0001: new disk, initializing [ 5023.633965] Lustre: Skipped 1 previous similar message [ 5023.636226] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5023.642100] Lustre: Skipped 1 previous similar message [ 5025.471703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5025.477396] Lustre: Skipped 1 previous similar message [ 5025.481266] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5025.550195] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5027.503741] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5034.452870] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5037.165355] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5045.714874] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 5045.821414] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 5049.835198] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5049.841216] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5049.855313] Lustre: Skipped 1 previous similar message [ 5049.857950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5049.860185] Lustre: Skipped 2 previous similar messages [ 5055.639213] LustreError: 262817:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5055.643595] LustreError: 262817:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5055.746162] Lustre: server umount lustre-MDT0000 complete [ 5058.175804] LustreError: 259289:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758738525 with bad export cookie 10254942860373462866 [ 5058.181845] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5058.189848] LustreError: 259289:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5058.409356] Lustre: server umount lustre-MDT0001 complete [ 5070.767026] Lustre: server umount lustre-OST0000 complete [ 5084.060294] Lustre: server umount lustre-OST0001 complete [ 5093.288236] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 5098.692560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5098.898542] LustreError: 264595:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5098.946545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5098.949591] Lustre: Skipped 3 previous similar messages [ 5100.957685] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5106.239249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5108.654717] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5110.638873] Lustre: 265703:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5114.785953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5117.096438] LustreError: 266061:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5117.103041] LustreError: 266061:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 5118.797555] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5123.572625] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 5124.089835] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5125.968955] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 5127.774576] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5133.033196] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5135.391378] Lustre: 267555:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5143.899552] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 5143.960101] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 5151.419374] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 14:30:17 (1758738617) [ 5165.540382] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5165.542413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5165.547535] Lustre: Skipped 4 previous similar messages [ 5165.549673] Lustre: Skipped 4 previous similar messages [ 5170.595936] LustreError: 268262:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 5170.612111] LustreError: 268262:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 5170.663112] LustreError: 266252:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5170.669498] LustreError: 266252:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 6 previous similar messages [ 5170.721643] Lustre: server umount lustre-MDT0000 complete [ 5173.433734] LustreError: 264574:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758738641 with bad export cookie 10254942860373469159 [ 5173.439181] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5173.440406] LustreError: 264574:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5179.775712] Lustre: server umount lustre-MDT0001 complete [ 5188.630477] Lustre: server umount lustre-OST0000 complete [ 5197.754985] Lustre: server umount lustre-OST0001 complete [ 5201.560091] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 5207.602867] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 5213.873123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5218.915877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5223.377487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5227.848662] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5238.196220] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 5244.232087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5244.282358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5244.409787] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5244.448713] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5244.492786] Lustre: lustre-MDT0000: new disk, initializing [ 5244.568442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5244.570695] Lustre: Skipped 3 previous similar messages [ 5244.586672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5247.233659] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5254.676294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5254.726960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5254.780483] Lustre: 271768:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5254.972503] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5257.523348] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5266.033761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5266.092559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5268.091496] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5270.697272] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5278.377403] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5278.430111] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5280.342808] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5284.276170] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5292.428379] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5295.115320] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5295.117430] Lustre: Skipped 1 previous similar message [ 5310.911731] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 14:32:56 (1758738776) [ 5316.067883] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5316.076608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5316.076728] Lustre: Skipped 5 previous similar messages [ 5316.082529] Lustre: Skipped 7 previous similar messages [ 5318.643033] LustreError: 275384:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5318.649813] LustreError: 275384:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 5318.760898] Lustre: server umount lustre-MDT0000 complete [ 5321.191852] LustreError: 271775:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5321.214687] LustreError: 271775:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 7 previous similar messages [ 5321.865770] LustreError: 272554:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758738789 with bad export cookie 10254942860373477034 [ 5321.871366] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5321.888368] LustreError: 272554:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5322.226784] Lustre: server umount lustre-MDT0001 complete [ 5335.605824] Lustre: server umount lustre-OST0000 complete [ 5348.547706] Lustre: server umount lustre-OST0001 complete [ 5363.155715] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 5370.827802] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 5377.836737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5382.837307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5388.416486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5394.144489] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5400.543452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5400.613783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5400.805821] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5400.829112] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5400.831714] Lustre: Skipped 4 previous similar messages [ 5400.905597] Lustre: lustre-MDT0000: new disk, initializing [ 5400.907721] Lustre: Skipped 3 previous similar messages [ 5401.033858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5401.037720] Lustre: Skipped 3 previous similar messages [ 5404.399380] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5412.217191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5412.273569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5412.346969] Lustre: 278839:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5412.569367] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5412.573882] Lustre: Skipped 2 previous similar messages [ 5415.438856] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5421.793305] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5425.161252] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5431.238152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5431.323532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5433.234824] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5437.158768] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5442.389445] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5443.725116] Lustre: Mounted lustre-client [ 5444.318338] LustreError: 281343:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff8df1d4981000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 5444.355308] Lustre: Unmounted lustre-client [ 5449.563364] Lustre: server umount lustre-OST0000 complete [ 5453.794651] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5453.803455] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5453.809446] Lustre: Skipped 2 previous similar messages [ 5453.814202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5461.552694] Lustre: server umount lustre-MDT0000 complete [ 5464.556291] LustreError: 279774:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5464.573974] LustreError: 279774:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 5464.866261] LustreError: 278834:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758738932 with bad export cookie 10254942860373478637 [ 5464.871473] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5464.872451] LustreError: 278834:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5475.483916] Lustre: server umount lustre-MDT0001 complete [ 5484.270945] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 5486.536401] Key type lgssc unregistered [ 5498.080469] LNet: 282622:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5498.094184] LNetError: 282622:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5575.910963] LNet: Removed LNI 192.168.203.142@tcp [ 5576.659217] Key type .llcrypt unregistered [ 5576.662465] Key type ._llcrypt unregistered [ 5594.489758] Key type ._llcrypt registered [ 5594.491708] Key type .llcrypt registered [ 5594.584625] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 5606.074179] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 5606.879237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5606.902221] alg: No test for adler32 (adler32-zlib) [ 5607.950997] Lustre: Lustre: Build Version: 2.16.58_51_g2f926bd [ 5608.107514] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 5609.768147] Key type lgssc registered [ 5610.697462] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5618.158876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5623.321349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5628.517054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5633.579234] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5635.116773] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 14:38:20 (1758739100) [ 5640.335813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5640.399515] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5640.420087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5641.707562] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5641.744319] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5641.808172] Lustre: lustre-MDT0000: new disk, initializing [ 5641.894560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5641.911737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5645.126090] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5653.826955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5653.920641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5653.983823] Lustre: 285797:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5654.037959] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5654.039796] Lustre: Skipped 1 previous similar message [ 5654.087592] Lustre: lustre-MDT0001: new disk, initializing [ 5654.205630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5654.241489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5654.255688] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5657.712251] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5663.379950] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5665.741802] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5669.971559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5670.046608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5670.232669] Lustre: lustre-OST0000: new disk, initializing [ 5670.239725] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5670.319792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5674.315946] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all [ 5676.086958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5676.109050] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5676.173326] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5680.156814] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5690.420447] Lustre: DEBUG MARKER: == conf-sanity test 200a: check CPU partitions =========== 14:39:16 (1758739156) [ 5695.458943] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5695.465422] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5695.484526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5696.483547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5696.486713] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5701.605054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5701.611208] Lustre: Skipped 4 previous similar messages [ 5706.208132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5706.297496] LustreError: 288436:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5706.377511] Lustre: server umount lustre-MDT0000 complete [ 5709.541045] LustreError: 285787:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758739177 with bad export cookie 15508242005579111474 [ 5709.558676] LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5709.564809] LustreError: 285787:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5709.699443] LustreError: 288638:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5709.702889] LustreError: 288638:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5709.951992] Lustre: server umount lustre-MDT0001 complete [ 5722.419598] LustreError: 288839:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5722.422910] LustreError: 288839:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5722.608544] Lustre: server umount lustre-OST0000 complete [ 5763.048722] Lustre: DEBUG MARKER: == conf-sanity test 200b: set CPU pattern using core selection ========================================================== 14:40:28 (1758739228) [ 5796.499587] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 5798.896400] Key type lgssc unregistered [ 5799.127237] LNet: 291420:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5799.134963] LNetError: 291420:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5800.166485] LNet: Removed LNI 192.168.203.142@tcp [ 5800.929962] Key type .llcrypt unregistered [ 5800.932508] Key type ._llcrypt unregistered [ 5860.960122] Key type ._llcrypt registered [ 5860.961537] Key type .llcrypt registered [ 5861.050728] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 5861.390567] Key type .llcrypt unregistered [ 5861.392207] Key type ._llcrypt unregistered [ 5886.099679] Key type ._llcrypt registered [ 5886.107286] Key type .llcrypt registered [ 5886.275375] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 5886.588290] Key type .llcrypt unregistered [ 5886.590407] Key type ._llcrypt unregistered [ 5949.893273] Key type ._llcrypt registered [ 5949.894904] Key type .llcrypt registered [ 5949.991875] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 5950.224339] Key type .llcrypt unregistered [ 5950.228648] Key type ._llcrypt unregistered [ 5971.507384] Key type ._llcrypt registered [ 5971.508805] Key type .llcrypt registered [ 5971.594498] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 5971.852917] Key type .llcrypt unregistered [ 5971.854238] Key type ._llcrypt unregistered [ 5995.564912] Key type ._llcrypt registered [ 5995.566253] Key type .llcrypt registered [ 5995.681178] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 5996.016842] Key type .llcrypt unregistered [ 5996.018706] Key type ._llcrypt unregistered [ 6059.015396] Key type ._llcrypt registered [ 6059.017037] Key type .llcrypt registered [ 6059.092961] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 6059.395330] Key type .llcrypt unregistered [ 6059.400610] Key type ._llcrypt unregistered [ 6086.653892] Key type ._llcrypt registered [ 6086.655517] Key type .llcrypt registered [ 6086.748452] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 6086.965666] Key type .llcrypt unregistered [ 6086.967793] Key type ._llcrypt unregistered [ 6106.116710] Key type ._llcrypt registered [ 6106.118168] Key type .llcrypt registered [ 6106.209246] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 6106.417582] Key type .llcrypt unregistered [ 6106.420355] Key type ._llcrypt unregistered [ 6144.798286] Key type ._llcrypt registered [ 6144.803263] Key type .llcrypt registered [ 6144.910297] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 6155.286491] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 6156.076706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6156.097326] alg: No test for adler32 (adler32-zlib) [ 6157.141367] Lustre: Lustre: Build Version: 2.16.58_51_g2f926bd [ 6157.346639] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 6158.992187] Key type lgssc registered [ 6159.740901] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6166.931698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6172.711908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6177.783499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6182.970649] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6184.384139] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5967 sec ========== 14:47:30 (1758739650) [ 6185.691021] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:47:31 (1758739651) === [ 6188.214278] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:47:34 (1758739654) === [ 6216.771806] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 6219.162717] Key type lgssc unregistered [ 6219.403760] LNet: 307344:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6219.409486] LNetError: 307344:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6219.422851] LNet: Removed LNI 192.168.203.142@tcp [ 6220.160836] Key type .llcrypt unregistered [ 6220.165264] Key type ._llcrypt unregistered