[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 405673225 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002258] x2apic enabled [ 0.003004] Switched APIC routing to physical x2apic. [ 0.004006] kvm-guest: setup PV IPIs [ 0.006557] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007010] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008005] pid_max: default: 32768 minimum: 301 [ 0.009052] LSM: Security Framework initializing [ 0.010033] Yama: becoming mindful. [ 0.010755] SELinux: Initializing. [ 0.011038] *** VALIDATE selinux *** [ 0.018591] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.022444] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.023099] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024062] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025053] *** VALIDATE tmpfs *** [ 0.026308] *** VALIDATE proc *** [ 0.027143] *** VALIDATE cgroup *** [ 0.028004] *** VALIDATE cgroup2 *** [ 0.029187] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.030114] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.031003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.032016] Spectre V2 : User space: Vulnerable [ 0.033003] Speculative Store Bypass: Vulnerable [ 0.036201] debug: unmapping init [mem 0xffffffffbb059000-0xffffffffbb060fff] [ 0.038123] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.039471] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.040009] ... version: 2 [ 0.040812] ... bit width: 48 [ 0.041006] ... generic registers: 4 [ 0.041743] ... value mask: 0000ffffffffffff [ 0.042005] ... max period: 00007fffffffffff [ 0.042997] ... fixed-purpose events: 3 [ 0.043756] ... event mask: 000000070000000f [ 0.044142] rcu: Hierarchical SRCU implementation. [ 0.046161] smp: Bringing up secondary CPUs ... [ 0.047414] x86: Booting SMP configuration: [ 0.048018] .... node #0, CPUs: #1 #2 #3 [ 0.050775] smp: Brought up 1 node, 4 CPUs [ 0.051749] smpboot: Max logical packages: 1 [ 0.052006] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.237055] node 0 deferred pages initialised in 184ms [ 0.239385] devtmpfs: initialized [ 0.240157] x86/mm: Memory block size: 128MB [ 0.241987] gcov: version magic: 0x41383552 [ 0.243160] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.245048] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.246193] pinctrl core: initialized pinctrl subsystem [ 0.247070] [ 0.247331] ************************************************************* [ 0.249005] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.250005] ** ** [ 0.251004] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.252004] ** ** [ 0.253004] ** This means that this kernel is built to expose internal ** [ 0.255005] ** IOMMU data structures, which may compromise security on ** [ 0.256004] ** your system. ** [ 0.257004] ** ** [ 0.258005] ** If you see this message and you are not debugging the ** [ 0.259004] ** kernel, report this immediately to your vendor! ** [ 0.260004] ** ** [ 0.262005] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.263004] ************************************************************* [ 0.264363] NET: Registered protocol family 16 [ 0.265246] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.266026] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.268031] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.270013] cpuidle: using governor menu [ 0.270823] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.272285] PCI: Using configuration type 1 for base access [ 0.273075] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.278090] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.279025] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.280105] cryptd: max_cpu_qlen set to 1000 [ 0.281919] ACPI: Added _OSI(Module Device) [ 0.283011] ACPI: Added _OSI(Processor Device) [ 0.283964] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.285007] ACPI: Added _OSI(Processor Aggregator Device) [ 0.287648] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.292382] ACPI: Interpreter enabled [ 0.293035] ACPI: PM: (supports S0 S3 S4 S5) [ 0.295008] ACPI: Using IOAPIC for interrupt routing [ 0.296049] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.299248] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.306542] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.308019] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.310008] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.312035] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.314998] acpiphp: Slot [2] registered [ 0.315882] acpiphp: Slot [3] registered [ 0.317045] acpiphp: Slot [4] registered [ 0.318043] acpiphp: Slot [5] registered [ 0.318905] acpiphp: Slot [6] registered [ 0.320049] acpiphp: Slot [7] registered [ 0.320840] acpiphp: Slot [8] registered [ 0.322054] acpiphp: Slot [9] registered [ 0.323075] acpiphp: Slot [10] registered [ 0.323959] acpiphp: Slot [11] registered [ 0.325061] acpiphp: Slot [12] registered [ 0.325942] acpiphp: Slot [13] registered [ 0.327044] acpiphp: Slot [14] registered [ 0.327946] acpiphp: Slot [15] registered [ 0.329071] acpiphp: Slot [16] registered [ 0.329910] acpiphp: Slot [17] registered [ 0.331050] acpiphp: Slot [18] registered [ 0.331949] acpiphp: Slot [19] registered [ 0.333046] acpiphp: Slot [20] registered [ 0.333903] acpiphp: Slot [21] registered [ 0.335069] acpiphp: Slot [22] registered [ 0.335806] acpiphp: Slot [23] registered [ 0.335835] acpiphp: Slot [24] registered [ 0.337037] acpiphp: Slot [25] registered [ 0.337762] acpiphp: Slot [26] registered [ 0.338039] acpiphp: Slot [27] registered [ 0.338724] acpiphp: Slot [28] registered [ 0.340048] acpiphp: Slot [29] registered [ 0.340800] acpiphp: Slot [30] registered [ 0.341036] acpiphp: Slot [31] registered [ 0.341813] PCI host bridge to bus 0000:00 [ 0.343009] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.344006] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.345007] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.346006] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.348007] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.349008] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.350053] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.352458] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.354306] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.360279] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.363030] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.364006] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.365005] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.366006] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.368216] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.369404] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.371019] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.372313] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.375006] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.382477] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.384857] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.387458] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.391008] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.394008] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.402007] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.408538] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.413007] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.417007] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.428013] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.437355] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.441007] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.444008] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.454008] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.458849] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.462008] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.466009] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.473006] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.479544] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.482994] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.485009] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.492006] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.497740] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.500000] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.503007] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.511915] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.517013] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.518177] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.520184] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.521164] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.522099] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.525050] iommu: Default domain type: Passthrough [ 0.526368] SCSI subsystem initialized [ 0.527061] ACPI: bus type USB registered [ 0.527835] usbcore: registered new interface driver usbfs [ 0.528028] usbcore: registered new interface driver hub [ 0.528925] usbcore: registered new device driver usb [ 0.530096] pps_core: LinuxPPS API ver. 1 registered [ 0.530980] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.533014] PTP clock support registered [ 0.534116] EDAC MC: Ver: 3.0.0 [ 0.535357] PCI: Using ACPI for IRQ routing [ 0.536561] NetLabel: Initializing [ 0.537006] NetLabel: domain hash size = 128 [ 0.537811] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.538058] NetLabel: unlabeled traffic allowed by default [ 0.540035] vgaarb: loaded [ 0.541178] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.542005] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.548217] clocksource: Switched to clocksource kvm-clock [ 0.614893] VFS: Disk quotas dquot_6.6.0 [ 0.615738] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.617070] *** VALIDATE ramfs *** [ 0.617721] *** VALIDATE hugetlbfs *** [ 0.618527] pnp: PnP ACPI init [ 0.620449] pnp: PnP ACPI: found 6 devices [ 0.632782] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.634413] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.635518] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.636618] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.637838] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.639013] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.640441] NET: Registered protocol family 2 [ 0.641722] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.644423] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.646260] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.649389] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.651056] TCP: Hash tables configured (established 65536 bind 65536) [ 0.652470] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.653973] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.655383] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.656855] NET: Registered protocol family 1 [ 0.658140] RPC: Registered named UNIX socket transport module. [ 0.659231] RPC: Registered udp transport module. [ 0.660054] RPC: Registered tcp transport module. [ 0.660890] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.662058] NET: Registered protocol family 44 [ 0.662865] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.663934] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.665029] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.666230] PCI: CLS 0 bytes, default 64 [ 0.667116] Unpacking initramfs... [ 1.621519] debug: unmapping init [mem 0xffff8f623cc54000-0xffff8f623ffbffff] [ 1.623657] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.624900] software IO TLB: mapped [mem 0x00000000a3200000-0x00000000a7200000] (64MB) [ 1.626336] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 1.998589] Initialise system trusted keyrings [ 2.000112] Key type blacklist registered [ 2.001225] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.006479] zbud: loaded [ 2.008225] *** VALIDATE nfs *** [ 2.008838] *** VALIDATE nfs4 *** [ 2.009656] pstore: using deflate compression [ 2.011239] Platform Keyring initialized [ 2.075910] NET: Registered protocol family 38 [ 2.077014] Key type asymmetric registered [ 2.077803] Asymmetric key parser 'x509' registered [ 2.079147] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.080828] io scheduler mq-deadline registered [ 2.082485] io scheduler kyber registered [ 2.084193] io scheduler bfq registered [ 2.086045] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.088760] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.091441] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.093989] ACPI: Power Button [PWRF] [ 2.149632] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.204877] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.314338] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.369574] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.488269] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.516446] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.545274] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.549372] Non-volatile memory driver v1.3 [ 2.550173] Linux agpgart interface v0.103 [ 2.567705] virtio_blk virtio1: [vda] 132880 512-byte logical blocks (68.0 MB/64.9 MiB) [ 2.569271] vda: detected capacity change from 0 to 68034560 [ 2.577842] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.579342] vdb: detected capacity change from 0 to 1073741824 [ 2.588316] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.589774] vdc: detected capacity change from 0 to 2621440000 [ 2.597399] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.598723] vdd: detected capacity change from 0 to 2621440000 [ 2.606889] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.608337] vde: detected capacity change from 0 to 4294967296 [ 2.616883] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.618396] vdf: detected capacity change from 0 to 4294967296 [ 2.621791] libphy: Fixed MDIO Bus: probed [ 2.625221] usbcore: registered new interface driver usbserial_generic [ 2.626405] usbserial: USB Serial support registered for generic [ 2.627531] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.629808] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.630732] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.632305] mousedev: PS/2 mouse device common for all mice [ 2.633716] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.634339] rtc_cmos 00:05: RTC can wake from S4 [ 2.637106] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.637162] rtc_cmos 00:05: registered as rtc0 [ 2.639417] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.640741] intel_pstate: CPU model not supported [ 2.642529] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.644606] hid: raw HID events driver (C) Jiri Kosina [ 2.645601] usbcore: registered new interface driver usbhid [ 2.646566] usbhid: USB HID core driver [ 2.647306] drop_monitor: Initializing network drop monitor service [ 2.648476] Initializing XFRM netlink socket [ 2.649492] NET: Registered protocol family 10 [ 2.651040] Segment Routing with IPv6 [ 2.651699] NET: Registered protocol family 17 [ 2.652866] mpls_gso: MPLS GSO support [ 2.656474] RAS: Correctable Errors collector initialized. [ 2.657463] AVX version of gcm_enc/dec engaged. [ 2.658263] AES CTR mode by8 optimization enabled [ 2.698100] sched_clock: Marking stable (2698079809, 0)->(3299387348, -601307539) [ 2.700031] registered taskstats version 1 [ 2.701084] Loading compiled-in X.509 certificates [ 2.702142] zswap: loaded using pool lzo/zbud [ 2.715757] Key type big_key registered [ 2.723531] Key type encrypted registered [ 2.724325] ima: No TPM chip found, activating TPM-bypass! [ 2.725335] ima: Allocated hash algorithm: sha1 [ 2.726132] ima: No architecture policies found [ 2.727050] evm: Initialising EVM extended attributes: [ 2.727911] evm: security.selinux [ 2.728470] evm: security.ima [ 2.728994] evm: security.capability [ 2.729694] evm: HMAC attrs: 0x1 [ 2.730923] rtc_cmos 00:05: setting system clock to 2025-08-20 13:10:21 UTC (1755695421) [ 2.734193] debug: unmapping init [mem 0xffffffffbc003000-0xffffffffbc1fffff] [ 2.735642] debug: unmapping init [mem 0xffffffffbad82000-0xffffffffbb058fff] [ 2.747161] Write protecting the kernel read-only data: 28672k [ 2.748896] debug: unmapping init [mem 0xffffffffb9403000-0xffffffffb95fffff] [ 2.750645] debug: unmapping init [mem 0xffffffffb9d14000-0xffffffffb9dfffff] [ 2.769759] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 2.773983] systemd[1]: Detected virtualization kvm. [ 2.775031] systemd[1]: Detected architecture x86-64. [ 2.775915] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 2.795052] systemd[1]: No hostname configured. [ 2.795912] systemd[1]: Set hostname to . [ 2.796923] random: systemd: uninitialized urandom read (16 bytes read) [ 2.798037] systemd[1]: Initializing machine ID from random generator. [ 2.847179] random: ln: uninitialized urandom read (6 bytes read) [ 2.903161] random: systemd: uninitialized urandom read (16 bytes read) [ 2.904685] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 2.907364] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 2.909239] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.354220] device-mapper: uevent: version 1.0.3 [ 3.355978] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 3.815552] virtio_net virtio0 ens2: renamed from eth0 [ 3.823703] random: fast init done [ 3.861423] scsi host0: ata_piix [ 3.871171] scsi host1: ata_piix [ 3.873590] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 3.875707] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 7.364550] dracut-initqueue[584]: RTNETLINK answers: File exists [ 8.233181] random: crng init done [ 8.234423] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 8.592335] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 9.493912] printk: systemd: 26 output lines suppressed due to ratelimiting [ 9.654075] SELinux: Disabled at runtime. [ 9.711432] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 9.717432] systemd[1]: Detected virtualization kvm. [ 9.718393] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 10.013479] systemd[1]: initrd-switch-root.service: Succeeded. [ 10.015215] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 10.017851] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 10.020117] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 10.022274] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 10.028086] systemd[1]: Starting Journal Service... Starting Journal Service... [ 10.033768] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ 10.123548] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Mounting Huge Pages File System... [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 10.370764] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 10.528625] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 10.605261] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 10.684178] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 10.696336] EDAC sbridge: Ver: 1.1.2 [ 11.840377] Key type dns_resolver registered [ 12.089067] NFS: Registering the id_resolver key type [ 12.090982] Key type id_resolver registered [ 12.092449] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg359-server login: [ 33.392587] libcfs: loading out-of-tree module taints kernel. [ 33.400304] Key type ._llcrypt registered [ 33.401121] Key type .llcrypt registered [ 33.432813] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 40.419624] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 40.902080] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 40.910833] alg: No test for adler32 (adler32-zlib) [ 41.861605] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 42.133324] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 43.735110] Key type lgssc registered [ 44.229667] Lustre: Echo OBD driver; http://www.lustre.org/ [ 49.733376] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 50.534202] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 54.130375] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 56.714376] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 59.074037] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 63.897356] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 67.852733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 67.872229] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 67.880720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 68.969066] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 68.979801] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 69.022438] Lustre: lustre-MDT0000: new disk, initializing [ 69.046799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 69.052694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 70.233526] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 75.092151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 75.119556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 75.143169] Lustre: 6463:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 75.155931] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 75.157461] Lustre: Skipped 1 previous similar message [ 75.192792] Lustre: lustre-MDT0001: new disk, initializing [ 75.210451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 75.217204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 75.219561] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 76.350631] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 78.511792] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 81.459227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 81.481387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 81.555555] Lustre: lustre-OST0000: new disk, initializing [ 81.557160] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 81.570906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 82.345880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 82.348163] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 82.367740] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 83.207942] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 88.261904] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 88.288731] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 88.319552] Lustre: lustre-OST0001: new disk, initializing [ 88.321081] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 88.336779] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 90.041103] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 95.689814] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 97.968219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 97.970581] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 97.979774] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 99.184179] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 105.806779] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing check_logdir /tmp/testlogs/ [ 107.156977] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing yml_node [ 108.773758] Lustre: DEBUG MARKER: Client: 2.16.57.97 [ 109.752175] Lustre: DEBUG MARKER: MDS: 2.16.57.97 [ 110.468303] Lustre: DEBUG MARKER: OSS: 2.16.57.97 [ 110.915899] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Aug 20 09:12:10 EDT 2025 [ 116.298617] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 116.843275] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 118.751708] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 118.752163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 118.755952] Lustre: Skipped 3 previous similar messages [ 118.757443] Lustre: Skipped 2 previous similar messages [ 123.871985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 123.873983] Lustre: Skipped 4 previous similar messages [ 124.088648] LustreError: 12100:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 124.113917] Lustre: server umount lustre-MDT0000 complete [ 125.182560] LustreError: 8360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755695543 with bad export cookie 2572940755937113459 [ 125.183843] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 125.185872] LustreError: 8360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 128.991650] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 128.991945] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 128.994593] Lustre: Skipped 1 previous similar message [ 131.039538] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 131.042344] Lustre: Skipped 2 previous similar messages [ 131.321260] LustreError: 12301:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 131.323388] LustreError: 12301:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 131.384296] Lustre: server umount lustre-MDT0001 complete [ 138.664423] LustreError: 12504:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 138.666170] LustreError: 12504:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 138.681613] Lustre: server umount lustre-OST0000 complete [ 154.079072] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 154.094730] LustreError: 12705:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 154.096709] LustreError: 12705:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 154.150129] Lustre: server umount lustre-OST0001 complete [ 156.012354] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 159.436020] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 162.540773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 164.742853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 166.663787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 168.764923] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 171.118641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 171.141689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 171.222160] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 171.233346] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 171.268843] Lustre: lustre-MDT0000: new disk, initializing [ 171.294483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 171.300758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 172.546162] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 176.373058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 176.394360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 176.423173] Lustre: 14872:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 176.520882] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 177.663046] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 180.922295] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 182.320496] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 184.548885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 184.568730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 184.660018] Lustre: lustre-OST0000: new disk, initializing [ 184.661337] Lustre: Skipped 1 previous similar message [ 184.663155] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 184.665934] Lustre: Skipped 2 previous similar messages [ 184.692636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 184.694538] Lustre: Skipped 1 previous similar message [ 185.786506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 185.788721] Lustre: Skipped 1 previous similar message [ 185.790503] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 185.824617] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 186.453960] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 189.454084] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 190.802220] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 190.885665] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 192.188738] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 192.266375] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 196.069201] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 196.070374] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 196.071871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 196.074436] Lustre: Skipped 1 previous similar message [ 198.958641] LustreError: 17779:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 198.960345] LustreError: 17779:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 199.024661] Lustre: server umount lustre-OST0000 complete [ 201.186143] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 201.191896] Lustre: Skipped 1 previous similar message [ 206.299030] Lustre: server umount lustre-MDT0000 complete [ 206.304522] LustreError: 14884:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 206.308656] LustreError: 14884:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 207.439104] LustreError: 14909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755695626 with bad export cookie 2572940755937114880 [ 207.442138] LustreError: 14909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 207.442355] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 207.495465] LustreError: 18182:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 207.497073] LustreError: 18182:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 207.567206] Lustre: server umount lustre-MDT0001 complete [ 209.112802] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 09:13:48 (1755695628) [ 214.802319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 214.959786] LustreError: 19315:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 214.981323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 216.249047] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 218.732087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 219.975790] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 221.532408] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 222.595439] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 223.204801] Lustre: Failing over lustre-MDT0000 [ 223.207919] LustreError: 19347:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 223.210227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 223.210679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 223.210683] Lustre: Skipped 3 previous similar messages [ 223.212869] LustreError: 19347:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 223.218053] Lustre: Skipped 1 previous similar message [ 223.224316] LustreError: 19347:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 223.305455] Lustre: server umount lustre-MDT0000 complete [ 228.319927] LustreError: 19333:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 228.327611] LustreError: 19333:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 233.440128] LustreError: 19315:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 233.444029] LustreError: 19315:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 236.057532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 236.093614] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 237.332843] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 240.389125] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 241.633363] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 241.635678] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 241.640323] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 242.341579] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 242.342333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 242.344878] Lustre: Skipped 1 previous similar message [ 242.346517] Lustre: Skipped 1 previous similar message [ 248.442079] LustreError: 21619:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 248.443824] LustreError: 21619:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 248.470309] Lustre: server umount lustre-MDT0000 complete [ 248.800917] LustreError: 19333:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 248.805029] LustreError: 19333:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 249.634086] LustreError: 21620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755695668 with bad export cookie 2572940755937116287 [ 249.636459] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 249.636976] LustreError: 21620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 249.766170] Lustre: server umount lustre-MDT0001 complete [ 252.248524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 252.419483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 252.421330] Lustre: Skipped 2 previous similar messages [ 253.599952] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 256.203383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 257.450125] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 258.960226] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 259.950304] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 260.588501] Lustre: Failing over lustre-MDT0000 [ 260.593306] LustreError: 22327:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 260.595952] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 260.596188] LustreError: 22327:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 260.598963] Lustre: Skipped 1 previous similar message [ 260.602276] LustreError: 22327:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 260.694689] Lustre: server umount lustre-MDT0000 complete [ 265.696143] LustreError: 23003:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 265.699932] LustreError: 23003:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 273.347527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 273.382333] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 274.575300] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 277.456424] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 278.497685] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 278.500724] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 278.503295] Lustre: Skipped 1 previous similar message [ 278.508247] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 281.377649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 281.379242] Lustre: Skipped 5 previous similar messages [ 285.432667] LustreError: 24600:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 285.434300] LustreError: 24600:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 285.454821] Lustre: server umount lustre-MDT0000 complete [ 286.598103] LustreError: 22322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755695705 with bad export cookie 2572940755937117211 [ 286.601444] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 286.603652] LustreError: 22322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 286.734988] Lustre: server umount lustre-MDT0001 complete [ 289.922752] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 09:15:09 (1755695709) [ 295.333921] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 299.167435] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 302.741146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 304.949089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 307.048793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 309.226484] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 312.596827] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 316.010537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 316.027768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 316.101239] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 316.111398] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 316.151817] Lustre: lustre-MDT0000: new disk, initializing [ 316.179188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 317.392522] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 321.304566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 321.321167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 321.348294] Lustre: 28243:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 321.427110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 321.428802] Lustre: Skipped 3 previous similar messages [ 321.440114] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 322.541575] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 325.658703] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 327.107693] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 329.288891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 329.309745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 330.751128] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 330.772689] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 330.778243] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 331.005823] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 333.844571] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 336.072924] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 346.087750] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 346.087939] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 346.092177] LustreError: Skipped 2 previous similar messages [ 346.095781] Lustre: Skipped 2 previous similar messages [ 346.096548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 346.100972] Lustre: Skipped 1 previous similar message [ 347.656843] Lustre: server umount lustre-OST0000 complete [ 355.716863] LustreError: 31066:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 355.720661] LustreError: 31066:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 355.756483] Lustre: server umount lustre-MDT0000 complete [ 356.321225] LustreError: 28252:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 356.330634] LustreError: 28252:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 357.394655] LustreError: 28235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755695776 with bad export cookie 2572940755937117610 [ 357.399575] LustreError: 28235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 357.400653] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 357.699905] Lustre: server umount lustre-MDT0001 complete [ 362.199094] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 363.527400] Key type lgssc unregistered [ 363.654436] LNet: 31796:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 363.658205] LNetError: 31796:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 363.669318] LNet: Removed LNI 192.168.203.159@tcp [ 364.009124] Key type .llcrypt unregistered [ 364.010547] Key type ._llcrypt unregistered [ 375.478121] Key type ._llcrypt registered [ 375.479937] Key type .llcrypt registered [ 375.539780] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 375.852362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 375.870132] alg: No test for adler32 (adler32-zlib) [ 376.754996] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 376.870339] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 378.479175] Key type lgssc registered [ 379.060584] Lustre: Echo OBD driver; http://www.lustre.org/ [ 383.141090] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 383.153768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 384.353846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 386.210130] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 390.428905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 390.640696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 392.330359] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 394.906241] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 396.714647] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 399.644142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 399.834980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 400.878048] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 402.402563] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 404.748103] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 416.213383] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 09:17:15 (1755695835) [ 417.119486] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 417.975981] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 421.277459] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 09:17:20 (1755695840) [ 424.436420] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 09:17:23 (1755695843) [ 424.864795] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 428.714316] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 428.718757] Lustre: Skipped 2 previous similar messages [ 429.532268] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 09:17:28 (1755695848) [ 433.314767] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 433.319774] Lustre: Skipped 4 previous similar messages [ 439.866370] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 09:17:38 (1755695858) [ 441.062356] Lustre: *** cfs_fail_loc=131b, val=2*** [ 442.788880] Lustre: *** cfs_fail_loc=131b, val=2*** [ 446.879692] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 09:17:45 (1755695865) [ 447.818213] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 447.821688] Lustre: Skipped 9 previous similar messages [ 453.907169] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 09:17:52 (1755695872) [ 461.285970] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 461.288067] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 461.288795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 461.294090] Lustre: Skipped 1 previous similar message [ 465.456368] LustreError: 39688:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 465.556443] Lustre: server umount lustre-OST0000 complete [ 471.522197] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 471.523189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 471.529231] Lustre: Skipped 1 previous similar message [ 471.533287] Lustre: Skipped 1 previous similar message [ 473.620289] LustreError: 39892:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 473.625883] LustreError: 39892:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 473.657280] Lustre: server umount lustre-MDT0000 complete [ 475.405306] LustreError: 33140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755695894 with bad export cookie 10565827836708270428 [ 475.411910] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 475.415097] LustreError: 33140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 475.521195] LustreError: 40094:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 475.524626] LustreError: 40094:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 475.618766] Lustre: server umount lustre-MDT0001 complete [ 480.199276] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 481.596805] Key type lgssc unregistered [ 481.739504] LNet: 40620:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 481.743435] LNetError: 40620:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 481.755420] LNet: Removed LNI 192.168.203.159@tcp [ 482.183774] Key type .llcrypt unregistered [ 482.185641] Key type ._llcrypt unregistered [ 489.460177] Key type ._llcrypt registered [ 489.461376] Key type .llcrypt registered [ 489.497866] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 496.489326] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 496.913022] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 496.923043] alg: No test for adler32 (adler32-zlib) [ 497.780823] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 497.865869] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 499.463194] Key type lgssc registered [ 499.922150] Lustre: Echo OBD driver; http://www.lustre.org/ [ 503.837929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 507.078082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 509.819074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 512.884649] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 516.189585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 516.224430] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 516.234387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 517.337211] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 517.351451] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 517.396736] Lustre: lustre-MDT0000: new disk, initializing [ 517.427203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 517.436729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 519.252319] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 525.007818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 525.050263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 525.107709] Lustre: 43710:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 525.149709] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 525.153318] Lustre: Skipped 1 previous similar message [ 525.201710] Lustre: lustre-MDT0001: new disk, initializing [ 525.253687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 525.274967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 525.281498] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 527.026987] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 530.792420] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 531.833404] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 533.870180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 533.894434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 533.977580] Lustre: lustre-OST0000: new disk, initializing [ 533.979251] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 534.003947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 535.719206] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 536.826619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 536.829790] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 536.865342] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 538.530561] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 539.690661] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 539.750493] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 540.942907] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 541.001482] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 542.181457] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 542.182339] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 542.186892] LustreError: Skipped 1 previous similar message [ 542.195447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 542.201348] Lustre: Skipped 1 previous similar message [ 547.300491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 547.305967] Lustre: Skipped 1 previous similar message [ 547.632229] LustreError: 46621:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 547.716407] Lustre: server umount lustre-OST0000 complete [ 550.880547] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 550.885075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 550.892064] Lustre: Skipped 1 previous similar message [ 550.895510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 552.417017] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 555.642776] LustreError: 46823:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 555.644688] LustreError: 46823:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 555.670992] Lustre: server umount lustre-MDT0000 complete [ 556.973183] LustreError: 44485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755695975 with bad export cookie 3310973361586258137 [ 556.977431] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 556.978166] LustreError: 44485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 557.040412] LustreError: 47025:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 557.043756] LustreError: 47025:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 557.116349] Lustre: server umount lustre-MDT0001 complete [ 561.035680] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 09:19:40 (1755695980) [ 563.381696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 563.571312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 565.244178] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 568.404761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 568.537797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 569.829526] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 572.016455] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 573.747396] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 576.963687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 577.077457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 579.593298] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 582.162702] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 595.196762] LustreError: 49943:0:(llog.c:430:llog_init_handle()) llog type is not specified! [ 595.657177] Lustre: Modifying parameter general.timeout=1 in log params [ 596.914253] Lustre: Modifying parameter general.timeout=4 in log params [ 596.916532] Lustre: Skipped 2 previous similar messages [ 599.350879] Lustre: Modifying parameter general.timeout=10 in log params [ 599.353578] Lustre: Skipped 5 previous similar messages [ 603.507911] Lustre: Modifying parameter general.timeout=20 in log params [ 603.510609] Lustre: Skipped 9 previous similar messages [ 611.709869] Lustre: Modifying parameter general.timeout=40 in log params [ 611.713316] Lustre: Skipped 19 previous similar messages [ 628.164963] Lustre: Modifying parameter general.timeout=78 in log params [ 628.168618] Lustre: Skipped 37 previous similar messages [ 656.399231] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 09:21:15 (1755696075) [ 657.802966] Lustre: 56844:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 659.520069] Lustre: 57041:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 659.525317] Lustre: 57041:0:(mgs_llog.c:1345:mgs_modify_param()) Skipped 1 previous similar message [ 660.738658] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 in log params [ 660.743752] Lustre: Skipped 57 previous similar messages [ 662.170754] Lustre: 57358:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 662.179761] Lustre: 57358:0:(mgs_llog.c:1345:mgs_modify_param()) Skipped 2 previous similar messages [ 674.370779] Lustre: 58708:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 674.373873] LustreError: 58708:0:(mgs_handler.c:1119:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 675.963716] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 09:21:34 (1755696094) [ 681.446955] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 681.447281] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 681.449768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 681.449785] Lustre: Skipped 2 previous similar messages [ 681.466961] Lustre: Skipped 2 previous similar messages [ 686.564005] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 686.567695] Lustre: Skipped 4 previous similar messages [ 687.179636] LustreError: 59404:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 687.183537] LustreError: 59404:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 687.223221] Lustre: server umount lustre-MDT0000 complete [ 688.991855] LustreError: 47723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755696107 with bad export cookie 3310973361586259019 [ 688.998126] LustreError: 47723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 688.998623] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 689.186591] Lustre: server umount lustre-MDT0001 complete [ 701.111212] LustreError: 59809:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 701.115098] LustreError: 59809:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 701.211209] Lustre: server umount lustre-OST0000 complete [ 704.186380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 705.973223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 707.656570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 709.590531] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 712.974881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 713.133400] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 713.282709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 715.064998] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 719.270326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 719.334643] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 719.353160] Lustre: 61257:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 721.312805] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 724.045665] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 725.479763] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 728.835908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 728.961417] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 731.595239] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 734.300543] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 735.128956] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 735.135347] Lustre: Skipped 17 previous similar messages [ 741.348959] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 741.349259] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 741.364551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 741.368334] Lustre: Skipped 1 previous similar message [ 744.044140] LustreError: 63716:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 744.047204] LustreError: 63716:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 744.137786] Lustre: server umount lustre-OST0000 complete [ 760.287173] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 760.341216] LustreError: 63924:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 760.345014] LustreError: 63924:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 760.379383] Lustre: server umount lustre-MDT0000 complete [ 761.820819] LustreError: 61248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755696180 with bad export cookie 3310973361586261462 [ 761.827098] LustreError: 61248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 761.827115] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 761.994503] Lustre: server umount lustre-MDT0001 complete [ 766.449853] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 767.973968] Key type lgssc unregistered [ 768.163815] LNet: 64656:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 768.168389] LNetError: 64656:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 768.179678] LNet: Removed LNI 192.168.203.159@tcp [ 768.706203] Key type .llcrypt unregistered [ 768.708501] Key type ._llcrypt unregistered [ 779.819864] Key type ._llcrypt registered [ 779.821469] Key type .llcrypt registered [ 779.887089] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 780.372879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 780.399797] alg: No test for adler32 (adler32-zlib) [ 781.257342] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 781.342801] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 782.935237] Key type lgssc registered [ 783.470108] Lustre: Echo OBD driver; http://www.lustre.org/ [ 786.559971] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 786.573330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 787.774678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 789.473471] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 792.997662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 793.229963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 794.607029] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 796.815657] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 798.196563] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 801.291757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 801.575781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 804.343720] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 806.665384] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 820.514168] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 823.775854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 823.777898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 823.781415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 828.384746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 828.386406] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 828.391802] Lustre: Skipped 1 previous similar message [ 829.644368] LustreError: 68726:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 829.689672] Lustre: server umount lustre-MDT0000 complete [ 831.203599] LustreError: 68372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755696249 with bad export cookie 6127682465512545096 [ 831.210385] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 831.278586] LustreError: 68931:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 831.280886] LustreError: 68931:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 831.355662] Lustre: server umount lustre-MDT0001 complete [ 843.256694] LustreError: 69132:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 843.261728] LustreError: 69132:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 843.371782] Lustre: server umount lustre-OST0000 complete [ 846.944596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 848.678695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 850.368602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 851.848918] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 855.294739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 855.448270] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 855.469190] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 855.645424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 857.520611] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 861.607199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 861.637875] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 861.647287] Lustre: 70582:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 861.704029] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 861.705881] Lustre: Skipped 2 previous similar messages [ 861.816471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 863.625259] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 866.217584] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 867.804710] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 870.900352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 871.019601] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 871.205681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 873.213007] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 875.602247] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 881.681330] Lustre: 72809:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 882.417528] Lustre: Modifying parameter general.jobid_var=disable in log params [ 888.801976] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 888.804424] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 888.805365] LustreError: Skipped 1 previous similar message [ 888.812183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 888.815169] Lustre: Skipped 1 previous similar message [ 890.094106] LustreError: 73058:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 890.096294] LustreError: 73058:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 890.186746] Lustre: server umount lustre-OST0000 complete [ 892.383995] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 892.386312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 892.389699] Lustre: Skipped 1 previous similar message [ 893.922875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 893.926818] Lustre: Skipped 2 previous similar messages [ 898.133125] LustreError: 73266:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 898.137258] LustreError: 73266:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 898.185322] Lustre: server umount lustre-MDT0000 complete [ 899.041783] LustreError: 71332:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 899.045472] LustreError: 71332:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 899.670839] LustreError: 70574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755696318 with bad export cookie 6127682465512545908 [ 899.677461] LustreError: 70574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 899.677919] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 899.867344] Lustre: server umount lustre-MDT0001 complete [ 904.178250] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 905.506797] Key type lgssc unregistered [ 905.655452] LNet: 73995:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 905.658548] LNetError: 73995:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 905.667531] LNet: Removed LNI 192.168.203.159@tcp [ 906.142747] Key type .llcrypt unregistered [ 906.144899] Key type ._llcrypt unregistered [ 917.915236] Key type ._llcrypt registered [ 917.916692] Key type .llcrypt registered [ 917.954159] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 918.234948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 918.244466] alg: No test for adler32 (adler32-zlib) [ 919.089951] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 919.167291] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 920.751254] Key type lgssc registered [ 921.348929] Lustre: Echo OBD driver; http://www.lustre.org/ [ 925.254291] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 925.267118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 926.484837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 928.168948] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 932.095104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 932.389234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 934.105103] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 936.413044] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 938.232913] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 941.384443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 941.654775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 944.352715] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 946.513944] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 959.706742] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 960.618720] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 960.622841] Lustre: Skipped 1 previous similar message [ 961.949134] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 961.952122] Lustre: Skipped 2 previous similar messages [ 963.969271] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 963.971356] Lustre: Skipped 5 previous similar messages [ 968.251699] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 968.253524] Lustre: Skipped 11 previous similar messages [ 976.483990] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 976.485849] Lustre: Skipped 28 previous similar messages [ 992.511277] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 992.515098] Lustre: Skipped 44 previous similar messages [ 997.126297] Lustre: 83099:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 998.410719] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 09:26:57 (1755696417) [ 999.199106] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1000.049780] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 09:26:58 (1755696418) [ 1003.492117] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1003.493237] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1003.496436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1003.500212] Lustre: Skipped 1 previous similar message [ 1007.343173] LustreError: 83391:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1007.443195] Lustre: server umount lustre-OST0000 complete [ 1013.730202] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1013.731136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1013.738640] Lustre: Skipped 1 previous similar message [ 1013.742279] Lustre: Skipped 1 previous similar message [ 1015.500401] LustreError: 83594:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1015.505213] LustreError: 83594:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1015.545748] Lustre: server umount lustre-MDT0000 complete [ 1017.263261] LustreError: 75339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755696436 with bad export cookie 5632596002290859406 [ 1017.268699] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1017.272671] LustreError: 75339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1017.370396] LustreError: 83799:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1017.373962] LustreError: 83799:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1017.465349] Lustre: server umount lustre-MDT0001 complete [ 1021.500153] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 1022.583428] Key type lgssc unregistered [ 1022.713273] LNet: 84325:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1022.715746] LNetError: 84325:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1022.724315] LNet: Removed LNI 192.168.203.159@tcp [ 1023.140430] Key type .llcrypt unregistered [ 1023.141925] Key type ._llcrypt unregistered [ 1024.517990] Key type ._llcrypt registered [ 1024.519619] Key type .llcrypt registered [ 1024.564621] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1026.122462] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules [ 1026.552944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1026.562171] alg: No test for adler32 (adler32-zlib) [ 1027.420046] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 1027.518718] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 1029.103177] Key type lgssc registered [ 1029.628536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1030.051315] LustreError: 85429:0:(lu_object.c:1390:lu_context_key_register()) cfs_race id 60d sleeping [ 1038.400505] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 1041.853262] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1051.392980] LustreError: 85429:0:(lu_object.c:1390:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1051.400662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1052.581665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1053.908181] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1057.017639] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 09:27:56 (1755696476) [ 1058.833966] LustreError: 86462:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1058.838725] LustreError: 86462:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1058.841619] LustreError: 86462:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 1058.902547] LustreError: 87228:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1058.983352] Lustre: server umount lustre-MDT0000 complete [ 1063.959537] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 1065.036312] Key type lgssc unregistered [ 1065.158404] LNet: 87908:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1065.160410] LNetError: 87908:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1065.169456] LNet: Removed LNI 192.168.203.159@tcp [ 1065.451482] Key type .llcrypt unregistered [ 1065.452401] Key type ._llcrypt unregistered [ 1071.744946] Key type ._llcrypt registered [ 1071.745912] Key type .llcrypt registered [ 1071.777158] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 1072.100992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1072.135321] alg: No test for adler32 (adler32-zlib) [ 1072.971690] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 1073.045954] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 1074.615223] Key type lgssc registered [ 1075.109499] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1077.658564] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1077.664438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1078.822966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1079.983027] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1082.770334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1082.992262] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1084.213626] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1085.872580] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1087.204103] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1089.725717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1089.914230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1091.606733] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1093.373946] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1130.190622] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 09:29:09 (1755696549) [ 1130.921918] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1131.715487] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 09:29:10 (1755696550) [ 1134.048844] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1134.053266] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1134.061074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1136.096329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1136.097913] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1139.085064] LustreError: 91989:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1139.123247] Lustre: server umount lustre-MDT0000 complete [ 1140.704935] LustreError: 89058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755696559 with bad export cookie 11309739146362228448 [ 1140.711232] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1140.711243] LustreError: 89058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1140.801199] LustreError: 92192:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1140.804427] LustreError: 92192:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1140.884176] Lustre: server umount lustre-MDT0001 complete [ 1152.820429] LustreError: 92393:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1152.824122] LustreError: 92393:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1152.992275] Lustre: server umount lustre-OST0000 complete [ 1158.321168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1158.498117] LustreError: 93023:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1158.503273] LustreError: 93023:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1158.543644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1159.636908] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1162.574065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1162.837139] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1164.521483] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1167.002783] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1168.645579] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1171.601871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1174.737555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1174.767684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1174.866748] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1174.871956] LustreError: 93012:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1174.890312] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1174.895774] LustreError: 94870:0:(tgt_mount.c:2460:server_fill_super()) Unable to start targets: -98 [ 1174.900217] LustreError: 94870:0:(tgt_mount.c:1954:server_put_super()) no obd lustre-OST0000 [ 1174.903952] LustreError: 94870:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 1174.976601] Lustre: server umount lustre-OST0000 complete [ 1174.979436] LustreError: 94870:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1178.911775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1178.997185] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1179.002264] LustreError: 93012:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1179.007636] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1179.011765] LustreError: 95331:0:(tgt_mount.c:2460:server_fill_super()) Unable to start targets: -98 [ 1179.015039] LustreError: 95331:0:(tgt_mount.c:1954:server_put_super()) no obd lustre-OST0000 [ 1179.017751] LustreError: 95331:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 1179.073904] Lustre: server umount lustre-OST0000 complete [ 1179.075708] LustreError: 95331:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1180.289964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1183.392815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1183.481229] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1183.484638] Lustre: Found index 0 for lustre-OST0000, updating log [ 1183.488297] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1183.516040] Lustre: lustre-OST0000: new disk, initializing [ 1183.518228] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1183.663960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1186.155500] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1187.897173] Lustre: Failing over lustre-OST0000 [ 1187.934147] LustreError: 96406:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1187.937761] LustreError: 96406:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1188.029646] Lustre: server umount lustre-OST0000 complete [ 1192.928731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1192.929549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1192.931642] Lustre: Skipped 2 previous similar messages [ 1192.933516] Lustre: Skipped 2 previous similar messages [ 1198.048426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1198.051391] Lustre: Skipped 2 previous similar messages [ 1203.168149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1203.169808] Lustre: Skipped 1 previous similar message [ 1204.191087] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1204.221822] LustreError: 96607:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1204.223832] LustreError: 96607:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1204.249582] Lustre: server umount lustre-MDT0000 complete [ 1205.423133] LustreError: 96609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755696624 with bad export cookie 11309739146362231178 [ 1205.426364] LustreError: 96609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1205.426381] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1215.970366] LustreError: 96814:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1215.972110] LustreError: 96814:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1216.034600] Lustre: server umount lustre-MDT0001 complete [ 1219.934278] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 09:30:38 (1755696638) [ 1225.221951] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 1228.647252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1228.785665] LustreError: 98175:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1228.812473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1229.959469] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1232.748924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1234.075756] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1235.044975] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1237.453889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1237.665434] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 1238.691293] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1239.649671] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1242.795970] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 1242.795984] Lustre: lustre-OST0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1242.800540] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 1243.053667] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1243.080071] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1243.111003] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1243.145730] Lustre: lustre-OST0001: new disk, initializing [ 1243.147314] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1243.911507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1243.916441] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1243.962360] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1245.082992] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1248.144542] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1255.740261] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1263.585250] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 1263.590086] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1263.597916] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1268.618881] LustreError: 101390:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 1268.623208] LustreError: 101390:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1268.653959] Lustre: server umount lustre-MDT0001 complete [ 1269.720325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1269.731841] LustreError: 98882:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1269.735968] LustreError: 98882:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1272.458193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1272.511649] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1272.514780] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1272.538475] Lustre: 98164:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1272.817217] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1272.819647] Lustre: Skipped 3 previous similar messages [ 1274.026652] LustreError: 102117:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1274.031635] LustreError: 102117:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1274.037069] LustreError: 98166:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1274.074671] LustreError: 102129:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.203.159@tcp: cfg command failed: rc = -17 [ 1274.077983] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1274.077983] [ 1274.200083] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1277.930812] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 1277.933409] Lustre: Skipped 1 previous similar message [ 1278.646742] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 09:31:37 (1755696697) [ 1285.522596] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 1293.287683] Lustre: 103986:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1302.782513] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1304.533776] Lustre: 105117:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1359.840267] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1359.841060] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1359.841334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1359.841339] Lustre: Skipped 4 previous similar messages [ 1359.849217] Lustre: Skipped 5 previous similar messages [ 1364.701009] LustreError: 105574:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 1364.704048] LustreError: 105574:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1364.740735] Lustre: server umount lustre-MDT0000 complete [ 1364.961384] LustreError: 100440:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1364.970590] LustreError: 100440:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 1366.427630] LustreError: 105475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755696785 with bad export cookie 11309739146362232018 [ 1366.429834] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1366.433085] LustreError: 105475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1366.602556] Lustre: server umount lustre-MDT0001 complete [ 1377.323078] Lustre: server umount lustre-OST0000 complete [ 1389.154268] Lustre: server umount lustre-OST0001 complete [ 1392.431597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1398.764801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1399.542451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1405.677393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1409.857556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1410.181155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1416.312944] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 1419.656526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1419.665224] Lustre: lustre-MDT0000: reset Object Index mappings [ 1419.834360] LustreError: 109048:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1419.860107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1421.082828] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1423.773387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1423.777555] Lustre: lustre-MDT0001: reset Object Index mappings [ 1425.201126] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1426.090297] Lustre: 110150:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1428.195237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1429.474070] LustreError: 110509:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1429.478062] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 1430.222369] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1432.895657] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1434.159303] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 1434.159818] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:226 to 0x280000400:257) [ 1434.160561] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 1434.919381] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1438.087448] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1439.262517] Lustre: 111992:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1458.462029] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 09:34:37 (1755696877) [ 1490.402753] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1490.403582] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1490.404387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1490.404392] Lustre: Skipped 3 previous similar messages [ 1490.413066] Lustre: Skipped 3 previous similar messages [ 1493.958937] LustreError: 112343:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1493.960777] LustreError: 112343:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 1493.992929] Lustre: server umount lustre-MDT0000 complete [ 1495.256442] LustreError: 111994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755696914 with bad export cookie 11309739146362278988 [ 1495.260943] LustreError: 111994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1495.264990] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1495.520060] LustreError: 109775:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1495.525755] LustreError: 109775:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 1501.592289] Lustre: server umount lustre-MDT0001 complete [ 1509.056750] Lustre: server umount lustre-OST0000 complete [ 1516.396991] Lustre: server umount lustre-OST0001 complete [ 1518.346717] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 1521.204433] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 1524.391260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1526.535264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1528.526052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1530.694058] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1533.107827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1533.129215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1533.224961] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1533.235771] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1533.272751] Lustre: lustre-MDT0000: new disk, initializing [ 1533.305345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1534.662034] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1537.723098] Lustre: server umount lustre-MDT0000 complete [ 1538.678647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1540.946290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1540.964945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1541.062538] Lustre: Found index 0 for lustre-MDT0000, updating log [ 1541.067495] Lustre: 116194:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1541.076257] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 1542.501468] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1544.204692] Lustre: server umount lustre-MDT0000 complete [ 1546.932666] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 09:36:06 (1755696966) [ 1547.462523] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 1548.056832] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 09:36:07 (1755696967) [ 1553.359776] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 1556.486960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1556.630295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1556.632168] Lustre: Skipped 5 previous similar messages [ 1557.916871] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1560.629928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1560.651277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1560.680308] Lustre: 118202:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1560.702511] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1560.704206] Lustre: Skipped 1 previous similar message [ 1560.737373] Lustre: lustre-MDT0001: new disk, initializing [ 1560.779974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1560.782868] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1561.904743] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1564.136971] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1566.085398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1566.110247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1566.192820] Lustre: lustre-OST0000: new disk, initializing [ 1566.194501] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1566.899757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1566.902517] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1566.935727] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1567.912873] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1572.075851] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1572.104930] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1573.721350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1573.726605] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1573.771219] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1574.270056] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1579.156641] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1580.385958] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1723.525702] Lustre: 126945:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 1723.528283] Lustre: 126945:0:(osd_handler.c:1962:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 1723.530217] Lustre: 126945:0:(osd_handler.c:1969:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 1723.532438] Lustre: 126945:0:(osd_handler.c:1979:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 1723.534815] Lustre: 126945:0:(osd_handler.c:1986:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 1723.536671] Lustre: 126945:0:(osd_handler.c:1993:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1795.477985] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:40:14 (1755697214) [ 1796.575917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1796.577869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1796.580735] Lustre: Skipped 2 previous similar messages [ 1796.582580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1796.585902] Lustre: Skipped 7 previous similar messages [ 1801.290954] LustreError: 127128:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 1801.293590] LustreError: 127128:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 1801.354147] Lustre: server umount lustre-MDT0000 complete [ 1802.493934] LustreError: 119368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755697221 with bad export cookie 11309739146362331838 [ 1802.498140] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1802.499091] LustreError: 119368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1804.256046] LustreError: 121370:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1804.261163] LustreError: 121370:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 1808.782467] Lustre: server umount lustre-MDT0001 complete [ 1810.068312] LustreError: 127530:0:(ldlm_resource.c:981:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x280000400:0x3:0x0].0x0 (ffff8f6181d85700) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 1816.190064] Lustre: server umount lustre-OST0000 complete [ 1816.543114] Lustre: 119724:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1755697219/real 1755697219] req@ffff8f618c44dc00 x1840981210316160/t0(0) o104->lustre-OST0001@192.168.203.59@tcp:15/16 lens 328/224 e 0 to 1 dl 1755697235 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 [ 1816.551071] Lustre: 119724:0:(client.c:2453:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1817.470569] Lustre: 126957:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (11/6s); client may timeout req@ffff8f618ffe9f80 x1840981210314624/t4294968362(0) o6->lustre-MDT0001-mdtlov_UUID@0@lo:600/0 lens 544/432 e 0 to 0 dl 1755697230 ref 1 fl Complete:/200/0 rc 0/0 job:'osp-syn-1-1.0' uid:0 gid:0 projid:4294967295 [ 1817.478969] Lustre: 126957:0:(service.c:2560:ptlrpc_server_handle_request()) Skipped 3 previous similar messages [ 1823.610960] Lustre: server umount lustre-OST0001 complete [ 1825.700232] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 1828.742132] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 1832.432082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1834.898824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1837.088444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1839.408891] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1841.822376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1841.847710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1841.941599] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1841.956208] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1841.958305] Lustre: Skipped 1 previous similar message [ 1841.994268] Lustre: lustre-MDT0000: new disk, initializing [ 1841.995684] Lustre: Skipped 1 previous similar message [ 1842.021881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1842.024473] Lustre: Skipped 3 previous similar messages [ 1842.030888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1843.454951] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1847.678806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1847.704915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1847.743247] Lustre: 129898:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1847.852778] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1849.188751] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1852.407712] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1853.623293] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1856.112397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1856.137297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1857.339115] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1857.372816] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1858.246862] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1861.442234] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1864.915437] Lustre: lustre-MDD0000: changelog on [ 1865.871167] Lustre: lustre-MDD0001: changelog on [ 2235.950224] Lustre: lustre-MDD0001: changelog off [ 2236.540696] Lustre: 129906:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 2236.830208] Lustre: lustre-MDD0000: changelog off [ 2241.507312] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2241.509354] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2241.510582] LustreError: Skipped 1 previous similar message [ 2241.511363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2241.514290] Lustre: Skipped 6 previous similar messages [ 2241.515606] Lustre: Skipped 8 previous similar messages [ 2244.529099] LustreError: 139095:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2244.532673] LustreError: 139095:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 2244.633983] Lustre: server umount lustre-OST0000 complete [ 2252.097741] Lustre: server umount lustre-MDT0000 complete [ 2253.384367] LustreError: 130673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755697672 with bad export cookie 11309739146363102622 [ 2253.387066] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2253.387718] LustreError: 130673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2253.547067] Lustre: server umount lustre-MDT0001 complete [ 2257.518917] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 2258.678735] Key type lgssc unregistered [ 2258.829810] LNet: 140022:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2258.834417] LNetError: 140022:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2258.844738] LNet: Removed LNI 192.168.203.159@tcp [ 2259.230132] Key type .llcrypt unregistered [ 2259.231330] Key type ._llcrypt unregistered [ 2265.706458] Key type ._llcrypt registered [ 2265.707574] Key type .llcrypt registered [ 2265.745709] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 2272.141901] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 2272.585484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2272.591828] alg: No test for adler32 (adler32-zlib) [ 2273.448792] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 2273.540165] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 2275.119141] Key type lgssc registered [ 2275.463778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2278.454155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2280.689190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2282.696422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2284.871654] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2287.131194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2287.152841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2287.157731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2288.246935] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2288.260237] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2288.300707] Lustre: lustre-MDT0000: new disk, initializing [ 2288.327553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2288.334856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2289.678534] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2294.036416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2294.058654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2294.092553] Lustre: 143159:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2294.113629] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2294.115549] Lustre: Skipped 1 previous similar message [ 2294.152381] Lustre: lustre-MDT0001: new disk, initializing [ 2294.190705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2294.208581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2294.213743] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2295.658022] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2298.820314] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2300.053034] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2302.324829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2302.346737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2302.449758] Lustre: lustre-OST0000: new disk, initializing [ 2302.452924] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2302.488137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2304.413529] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2307.610295] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2312.184970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2312.188161] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2312.225248] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2315.379850] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_module obdecho/obdecho [ 2316.010889] LustreError: 145758:0:(echo_client.c:670:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 2316.015357] LustreError: 145758:0:(obd_config.c:779:class_setup()) setup ec failed (-22) [ 2316.625840] LustreError: 145855:0:(echo_client.c:1880:echo_md_handler()) Can not resolve the path : rc = -2 [ 2322.404397] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2322.406103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2322.411639] Lustre: Skipped 2 previous similar messages [ 2322.415147] Lustre: Skipped 1 previous similar message [ 2326.076020] LustreError: 146195:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2326.100565] Lustre: server umount lustre-MDT0000 complete [ 2327.455924] LustreError: 143148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755697746 with bad export cookie 7638390458586834164 [ 2327.460810] LustreError: 143148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2327.461057] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2327.522539] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2327.533188] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2327.536482] Lustre: Skipped 1 previous similar message [ 2327.551380] LustreError: 146397:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2327.554110] LustreError: 146397:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2327.634551] Lustre: server umount lustre-MDT0001 complete [ 2335.022253] LustreError: 146597:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2335.024917] LustreError: 146597:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2335.109134] Lustre: server umount lustre-OST0000 complete [ 2342.954512] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 2346.419748] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 2349.751704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2352.042407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2354.125934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2356.279047] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2358.687050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2358.707557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2358.784790] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2358.795331] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2358.830448] Lustre: lustre-MDT0000: new disk, initializing [ 2358.855850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2358.863207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2360.121859] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2364.457584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2364.476800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2364.513603] Lustre: 149527:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2364.539429] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2364.541027] Lustre: Skipped 1 previous similar message [ 2364.578971] Lustre: lustre-MDT0001: new disk, initializing [ 2364.614678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2364.626536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2364.631566] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2365.972760] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2369.129592] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2370.383934] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2372.628706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2372.647728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2372.734257] Lustre: lustre-OST0000: new disk, initializing [ 2372.735971] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2372.766618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2374.328192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2374.330904] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2374.362574] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2374.581093] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2377.684267] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2379.135576] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2379.217655] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2380.562900] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2380.650337] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2383.340152] LustreError: 152435:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2383.341879] LustreError: 152435:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2383.417206] Lustre: server umount lustre-OST0000 complete [ 2384.865340] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2384.865858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2384.869424] Lustre: Skipped 1 previous similar message [ 2384.875202] Lustre: Skipped 1 previous similar message [ 2389.985276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2389.988706] Lustre: Skipped 1 previous similar message [ 2395.105148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2395.106815] Lustre: Skipped 1 previous similar message [ 2395.108081] LustreError: 152635:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2395.109813] LustreError: 152635:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2395.133932] Lustre: server umount lustre-MDT0000 complete [ 2396.358027] LustreError: 151259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755697815 with bad export cookie 7638390458586835109 [ 2396.360951] LustreError: 151259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2396.361804] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2406.883093] LustreError: 152839:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2406.884855] LustreError: 152839:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2406.980117] Lustre: server umount lustre-MDT0001 complete [ 2408.175328] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 09:50:27 (1755697827) [ 2408.732633] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 2409.329318] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 09:50:28 (1755697828) [ 2411.942744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2412.086733] LustreError: 153510:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2412.093369] LustreError: 153510:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2412.107961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2413.351465] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2416.192509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2417.743498] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2419.579370] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2420.955946] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2423.331774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2425.688464] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2427.686586] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2442.211225] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2442.212729] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2442.212884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2442.216420] Lustre: Skipped 2 previous similar messages [ 2446.272389] LustreError: 155760:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2446.274265] LustreError: 155760:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2446.304929] Lustre: server umount lustre-MDT0000 complete [ 2447.330886] LustreError: 155649:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2447.335707] LustreError: 155649:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2447.564072] LustreError: 153489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755697866 with bad export cookie 7638390458586835977 [ 2447.567129] LustreError: 153489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2447.567327] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2448.472717] LustreError: 155649:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2452.447827] LustreError: 155649:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2452.448158] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2453.785620] Lustre: server umount lustre-MDT0001 complete [ 2456.200053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2461.210252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2461.300657] LustreError: 157022:0:(llog.c:1602:llog_backup()) MGC192.168.203.159@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 2461.304812] Lustre: 157022:0:(mgc_request_server.c:726:mgc_llog_local_copy()) MGC192.168.203.159@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 2472.928547] LustreError: 157030:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2472.933551] LustreError: 157030:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 2483.168575] LustreError: 157030:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2483.175696] LustreError: 157030:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 2500.696584] LustreError: 157048:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2500.704962] LustreError: 157048:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 15 previous similar messages [ 2522.591383] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 2522.594050] Lustre: 157022:0:(mgc_request.c:1876:mgc_process_log()) MGC192.168.203.159@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 2532.832924] LustreError: 157048:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2532.842106] LustreError: 157048:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 19 previous similar messages [ 2549.727285] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 2549.730302] Lustre: 153501:0:(mgc_request.c:1876:mgc_process_log()) MGC192.168.203.159@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 2563.558578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2563.560712] Lustre: Skipped 2 previous similar messages [ 2564.784330] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2567.654522] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 2568.113231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2569.759224] LustreError: 153501:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2593.254150] Lustre: Evicted from MGS (at 192.168.203.159@tcp) after server handle changed from 0x0 to 0x6a0103a6f3cd4554 [ 2593.257192] Lustre: MGC192.168.203.159@tcp: Connection restored to (at 0@lo) [ 2594.666762] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2596.533411] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2597.074324] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2598.266910] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2598.374045] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 2601.894803] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2607.133173] LustreError: 158496:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2607.135497] LustreError: 158496:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2607.165178] Lustre: server umount lustre-MDT0000 complete [ 2608.411489] LustreError: 153490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698027 with bad export cookie 7638390458586842452 [ 2608.416326] LustreError: 153490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2608.416688] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2608.609216] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2608.610116] LustreError: 157028:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2608.615380] Lustre: Skipped 1 previous similar message [ 2608.622766] LustreError: 157028:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 43 previous similar messages [ 2608.629519] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2608.632449] Lustre: Skipped 6 previous similar messages [ 2614.634271] Lustre: server umount lustre-MDT0001 complete [ 2616.519687] Lustre: server umount lustre-OST0000 complete [ 2624.405624] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 2627.650265] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 2631.329577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2633.785667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2635.839673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2638.128484] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2640.803996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2640.845432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2640.980047] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2641.003862] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2641.063175] Lustre: lustre-MDT0000: new disk, initializing [ 2641.104655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2641.110105] Lustre: Skipped 1 previous similar message [ 2641.121431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2642.558465] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2647.298384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2647.325457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2647.356636] Lustre: 161827:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2647.473369] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2649.071926] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2652.623736] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2653.974240] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2656.786825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2656.815121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2659.326486] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2662.398973] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2662.436410] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2663.081189] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2664.817610] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2664.922802] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2666.473108] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2666.542617] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2667.494289] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2667.494900] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2667.496151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2667.496159] Lustre: Skipped 1 previous similar message [ 2667.511940] Lustre: Skipped 1 previous similar message [ 2669.378274] Lustre: server umount lustre-OST0000 complete [ 2676.861163] LustreError: 164939:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2676.862965] LustreError: 164939:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 2676.893211] Lustre: server umount lustre-MDT0000 complete [ 2678.413306] LustreError: 163564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698097 with bad export cookie 7638390458586859266 [ 2678.418888] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2678.419414] LustreError: 163564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2678.572595] Lustre: server umount lustre-MDT0001 complete [ 2681.993330] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 09:55:01 (1755698101) [ 2684.959994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2686.555396] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2690.185247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2692.087028] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2694.626886] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2696.433451] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2699.592777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2702.051897] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2704.288620] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2719.381475] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 09:55:38 (1755698138) [ 2721.249105] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2721.251419] LustreError: Skipped 1 previous similar message [ 2721.252613] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2721.255758] Lustre: Skipped 2 previous similar messages [ 2725.350187] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2726.708967] Lustre: server umount lustre-OST0000 complete [ 2735.584475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2735.587685] Lustre: Skipped 9 previous similar messages [ 2742.239111] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2742.318249] Lustre: server umount lustre-MDT0000 complete [ 2743.778343] LustreError: 167418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698162 with bad export cookie 7638390458586860148 [ 2743.784392] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2743.785389] LustreError: 167418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2743.956520] Lustre: server umount lustre-MDT0001 complete [ 2748.265776] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 2749.290549] Key type lgssc unregistered [ 2749.413605] LNet: 169188:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2749.418500] LNetError: 169188:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2749.427654] LNet: Removed LNI 192.168.203.159@tcp [ 2749.919463] Key type .llcrypt unregistered [ 2749.921279] Key type ._llcrypt unregistered [ 2758.912142] Key type ._llcrypt registered [ 2758.913519] Key type .llcrypt registered [ 2758.965117] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 2759.389508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2759.397576] alg: No test for adler32 (adler32-zlib) [ 2760.266824] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 2760.353890] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 2761.927281] Key type lgssc registered [ 2762.504235] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2765.689131] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2765.695766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2782.175527] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2788.319252] LustreError: 170382:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.159@tcp: failed processing log, type 4: rc = -110 [ 2794.463507] LustreError: 170382:0:(llog_osd.c:243:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 2794.471496] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 2794.475427] LustreError: MGC192.168.203.159@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 2794.483225] LustreError: 170382:0:(tgt_mount.c:1710:server_start_targets()) failed to start server lustre-OST0000: -22 [ 2794.493379] LustreError: 170382:0:(tgt_mount.c:2460:server_fill_super()) Unable to start targets: -22 [ 2794.500820] LustreError: 170382:0:(tgt_mount.c:1954:server_put_super()) no obd lustre-OST0000 [ 2794.508448] LustreError: 170382:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2794.585429] Lustre: server umount lustre-OST0000 complete [ 2794.587402] LustreError: 170382:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 2798.709805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2799.882229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2801.582965] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2805.512750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2805.647131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2807.147921] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2809.258472] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2810.906512] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2813.387623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2813.452955] LustreError: 172461:0:(llog_osd.c:243:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 2813.456075] Lustre: 172461:0:(mgc_request_server.c:706:mgc_llog_local_copy()) MGC192.168.203.159@tcp: invalid local config log lustre-OST0000: rc = -22 [ 2813.501058] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2815.335092] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2817.376128] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2821.089242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2821.091866] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2821.096188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2821.104492] Lustre: Skipped 2 previous similar messages [ 2821.107826] Lustre: Skipped 1 previous similar message [ 2824.261134] LustreError: 173104:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2824.264156] LustreError: 173104:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2824.297314] Lustre: server umount lustre-MDT0000 complete [ 2825.841060] LustreError: 170863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698244 with bad export cookie 2841693698329977733 [ 2825.846763] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2825.847822] LustreError: 170863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2825.935647] LustreError: 173308:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2825.939235] LustreError: 173308:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2826.029561] Lustre: server umount lustre-MDT0001 complete [ 2837.939969] LustreError: 173509:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2837.943906] LustreError: 173509:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2838.031713] Lustre: server umount lustre-OST0000 complete [ 2841.941418] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 2846.329743] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 2850.430753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2853.273684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2856.181708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2858.883299] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2861.992300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2862.032720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2862.174202] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2862.188758] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2862.246428] Lustre: lustre-MDT0000: new disk, initializing [ 2862.284156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2862.295403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2864.082796] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2869.185635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2869.228158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2869.285954] Lustre: 175830:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2869.329398] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2869.333569] Lustre: Skipped 1 previous similar message [ 2869.385412] Lustre: lustre-MDT0001: new disk, initializing [ 2869.434614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2869.459710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2869.468868] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2871.021591] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2875.079198] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2876.800573] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2879.936109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2879.970210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2880.110057] Lustre: lustre-OST0000: new disk, initializing [ 2880.113170] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2880.154315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2880.836574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2880.842995] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2880.898564] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2882.665148] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2886.013846] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2887.783059] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2887.876077] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2889.352044] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2889.483356] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2891.235426] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2891.237650] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2891.239550] LustreError: Skipped 1 previous similar message [ 2891.246887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2891.251744] Lustre: Skipped 2 previous similar messages [ 2896.357195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2896.361507] Lustre: Skipped 1 previous similar message [ 2896.371221] LustreError: 178735:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2896.377970] LustreError: 178735:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2896.477172] Lustre: server umount lustre-OST0000 complete [ 2900.448401] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2900.452997] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2900.460091] Lustre: Skipped 1 previous similar message [ 2900.463188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2904.402142] LustreError: 178938:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2904.405759] LustreError: 178938:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2904.448358] Lustre: server umount lustre-MDT0000 complete [ 2906.027724] LustreError: 177563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698324 with bad export cookie 2841693698329978419 [ 2906.030839] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2906.031387] LustreError: 177563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2906.164406] Lustre: server umount lustre-MDT0001 complete [ 2909.775354] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 09:58:48 (1755698328) [ 2916.921966] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 2918.264365] Key type lgssc unregistered [ 2918.410448] LNet: 180318:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2918.412920] LNetError: 180318:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2918.423486] LNet: Removed LNI 192.168.203.159@tcp [ 2918.832774] Key type .llcrypt unregistered [ 2918.834400] Key type ._llcrypt unregistered [ 2927.116141] Key type ._llcrypt registered [ 2927.117302] Key type .llcrypt registered [ 2927.159520] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 2927.470322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2927.476804] alg: No test for adler32 (adler32-zlib) [ 2928.339587] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 2928.419451] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 2929.999218] Key type lgssc registered [ 2930.371196] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2933.805902] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2933.817734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2935.006760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2936.538945] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2940.608111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2940.772601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2942.568773] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2944.977382] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2946.452180] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2949.711186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2949.777711] Lustre: *** cfs_fail_loc=131e, val=0*** [ 2949.778905] LustreError: 183115:0:(llog.c:1642:llog_backup()) MGC192.168.203.159@tcp: failed to backup log lustre-OST0000: rc = -28 [ 2949.781063] Lustre: 183115:0:(mgc_request_server.c:703:mgc_llog_local_copy()) MGC192.168.203.159@tcp: can't backup local config lustre-OST0000: rc = -28 [ 2949.814968] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2952.202644] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2954.490146] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2956.258291] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2956.262134] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2956.265340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2956.271730] Lustre: Skipped 2 previous similar messages [ 2956.275496] Lustre: Skipped 1 previous similar message [ 2961.378593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2961.382557] Lustre: Skipped 3 previous similar messages [ 2961.487637] LustreError: 183759:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2961.522310] Lustre: server umount lustre-MDT0000 complete [ 2963.264957] LustreError: 181470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698382 with bad export cookie 6849703543559603065 [ 2963.271617] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2963.271946] LustreError: 181470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2963.367991] LustreError: 183963:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2963.371514] LustreError: 183963:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2963.465826] Lustre: server umount lustre-MDT0001 complete [ 2975.221175] LustreError: 184163:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2975.224955] LustreError: 184163:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2975.322199] Lustre: server umount lustre-OST0000 complete [ 2979.448166] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 2983.060056] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 2987.814952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2990.880461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2993.511058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2996.185068] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2999.120787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2999.147438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2999.257125] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2999.274871] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2999.326086] Lustre: lustre-MDT0000: new disk, initializing [ 2999.358017] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2999.365914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3000.898540] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3006.141285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3006.179397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3006.228601] Lustre: 186486:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3006.261567] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3006.263932] Lustre: Skipped 1 previous similar message [ 3006.266523] LustreError: 187246:0:(llog.c:1642:llog_backup()) MGC192.168.203.159@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 3006.272100] LustreError: 187246:0:(llog.c:1642:llog_backup()) Skipped 1 previous similar message [ 3006.275891] Lustre: 187246:0:(mgc_request_server.c:726:mgc_llog_local_copy()) MGC192.168.203.159@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 3006.288139] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3006.291817] Lustre: Skipped 1 previous similar message [ 3006.343416] Lustre: lustre-MDT0001: new disk, initializing [ 3006.389802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3006.408379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3006.414042] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3007.911776] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3011.637249] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3013.287822] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3016.073334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3016.108637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3016.215754] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3016.217060] Lustre: Skipped 1 previous similar message [ 3016.218348] LustreError: 188196:0:(llog.c:1642:llog_backup()) MGC192.168.203.159@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3016.220962] LustreError: 188196:0:(llog.c:1642:llog_backup()) Skipped 1 previous similar message [ 3016.222847] Lustre: 188196:0:(mgc_request_server.c:726:mgc_llog_local_copy()) MGC192.168.203.159@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 3016.225653] Lustre: 188196:0:(mgc_request_server.c:726:mgc_llog_local_copy()) Skipped 1 previous similar message [ 3016.233244] Lustre: lustre-OST0000: new disk, initializing [ 3016.234933] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3016.271768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3018.744029] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3022.336937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3022.343523] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3022.395238] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3022.580360] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3024.142958] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3024.212849] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3025.591688] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3025.682794] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3027.425541] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3027.426961] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3027.429685] LustreError: Skipped 1 previous similar message [ 3027.434512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3027.437674] Lustre: Skipped 1 previous similar message [ 3028.397125] LustreError: 189396:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3028.400913] LustreError: 189396:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3028.460958] Lustre: server umount lustre-OST0000 complete [ 3032.032241] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3032.036142] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3032.041490] Lustre: Skipped 1 previous similar message [ 3032.044123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3037.664828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3037.668472] Lustre: Skipped 2 previous similar messages [ 3044.319103] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3044.349616] LustreError: 189597:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3044.352745] LustreError: 189597:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3044.384982] Lustre: server umount lustre-MDT0000 complete [ 3046.014527] LustreError: 188220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698464 with bad export cookie 6849703543559603772 [ 3046.017887] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3046.018973] LustreError: 188220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3046.199962] Lustre: server umount lustre-MDT0001 complete [ 3049.677956] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 10:01:08 (1755698468) [ 3056.214609] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 3060.373790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3060.582297] LustreError: 191171:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3060.590692] LustreError: 191171:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3060.602793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3062.191650] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3065.825231] LustreError: 191171:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3065.992877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3067.728849] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3069.094840] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3072.052863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3074.722787] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3078.097508] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3078.134787] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3078.217644] Lustre: lustre-OST0001: new disk, initializing [ 3078.220543] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3080.894152] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3081.562188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3081.568229] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3081.621482] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3087.262632] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3094.419497] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3100.299348] Lustre: DEBUG MARKER: ADD OST3 [ 3102.501109] loop: module loaded [ 3102.531542] loop0: detected capacity change from 0 to 204800000 [ 3102.669030] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3102.673813] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3102.680979] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3102.739447] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3105.277286] loop0: detected capacity change from 0 to 204800000 [ 3107.891455] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3107.917863] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3107.989296] Lustre: lustre-OST0003: new disk, initializing [ 3107.991582] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 3108.042438] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 3108.042994] Lustre: DEBUG MARKER: STOP OST3 [ 3108.045580] Lustre: Skipped 3 previous similar messages [ 3108.929944] Lustre: Failing over lustre-OST0003 [ 3108.953309] LustreError: 195379:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 3108.955081] LustreError: 195379:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3108.975654] Lustre: server umount lustre-OST0003 complete [ 3109.065315] LustreError: 193029:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3110.596710] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3114.317159] Lustre: DEBUG MARKER: START OST3 again [ 3114.472103] LustreError: 193029:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3114.482490] LustreError: 193029:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 3117.574844] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3119.233820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 3119.242544] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 3119.264941] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 3120.195044] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3134.711934] Lustre: Failing over lustre-OST0003 [ 3134.743177] LustreError: 197426:0:(obd_class.h:479:obd_check_dev()) Device 33 not setup [ 3134.748179] LustreError: 197426:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3134.773790] Lustre: server umount lustre-OST0003 complete [ 3134.944962] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3134.945109] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3134.949362] LustreError: Skipped 1 previous similar message [ 3134.956346] Lustre: Skipped 1 previous similar message [ 3134.960736] LustreError: 192614:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3134.971209] LustreError: 192614:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3138.019092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3138.022754] Lustre: Skipped 2 previous similar messages [ 3143.390252] Lustre: server umount lustre-MDT0000 complete [ 3143.649136] LustreError: 191170:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3143.656957] LustreError: 191170:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 6 previous similar messages [ 3145.045069] LustreError: 192261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698563 with bad export cookie 6849703543559604640 [ 3145.051911] LustreError: 192261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3145.053417] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3148.770490] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3148.778879] Lustre: Skipped 5 previous similar messages [ 3151.418060] Lustre: server umount lustre-MDT0001 complete [ 3153.021517] Lustre: server umount lustre-OST0000 complete [ 3154.407443] Lustre: server umount lustre-OST0001 complete [ 3156.638826] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 3159.696778] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 3163.504887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3166.580934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3169.073776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3171.747064] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3174.618932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3174.655559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3174.784204] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3174.799076] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3174.846666] Lustre: lustre-MDT0000: new disk, initializing [ 3174.880856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3174.884235] Lustre: Skipped 1 previous similar message [ 3174.893744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3176.346986] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3181.401029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3181.422744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3181.451168] Lustre: 200446:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3181.542894] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3183.049921] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3186.992104] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3188.573821] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3191.569041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3191.592970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3191.676880] Lustre: lustre-OST0000: new disk, initializing [ 3191.678120] Lustre: Skipped 1 previous similar message [ 3191.679642] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3191.681043] Lustre: Skipped 2 previous similar messages [ 3193.211831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3193.216129] Lustre: Skipped 1 previous similar message [ 3193.219295] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3193.269621] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3194.148536] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3197.385378] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3198.876216] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3198.954245] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3200.268720] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3200.355226] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3203.553395] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3203.554280] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3203.557093] LustreError: Skipped 1 previous similar message [ 3203.560819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3203.564087] Lustre: Skipped 8 previous similar messages [ 3207.282168] LustreError: 203353:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3207.284023] LustreError: 203353:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 3207.373017] Lustre: server umount lustre-OST0000 complete [ 3212.256101] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3212.259240] LustreError: Skipped 1 previous similar message [ 3214.951177] Lustre: server umount lustre-MDT0000 complete [ 3216.164939] LustreError: 200436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698634 with bad export cookie 6849703543559606901 [ 3216.167853] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3216.168329] LustreError: 200436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3216.297694] Lustre: server umount lustre-MDT0001 complete [ 3217.605637] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 10:03:56 (1755698636) [ 3223.511996] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 3226.923454] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 3230.621557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3233.613692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3235.697383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3238.142446] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3240.542938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3240.568127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3240.656895] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3240.674641] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3240.726196] Lustre: lustre-MDT0000: new disk, initializing [ 3240.755135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3242.289838] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3246.848134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3246.875399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3246.907542] Lustre: 206584:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3246.998682] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3248.213528] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3251.453175] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3252.557713] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3254.924117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3254.948464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3256.765917] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3256.796336] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3259.598115] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3260.828641] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3260.894188] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3262.019379] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3262.085364] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3267.041443] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3267.044024] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3267.044032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3267.044038] Lustre: Skipped 3 previous similar messages [ 3267.044764] Lustre: Skipped 4 previous similar messages [ 3268.839162] Lustre: server umount lustre-OST0000 complete [ 3276.091938] LustreError: 209694:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3276.093925] LustreError: 209694:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 3276.120886] Lustre: server umount lustre-MDT0000 complete [ 3277.247100] LustreError: 206575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698696 with bad export cookie 6849703543559607783 [ 3277.250509] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3277.252013] LustreError: 206575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3277.388875] Lustre: server umount lustre-MDT0001 complete [ 3280.003086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3280.151945] LustreError: 210370:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3280.160494] LustreError: 210370:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 3281.371339] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3284.920140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3286.768058] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3289.070700] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3290.596692] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3293.723243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3295.915578] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3297.756351] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3315.680343] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3315.682683] LustreError: Skipped 1 previous similar message [ 3315.684018] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3315.687823] Lustre: Skipped 2 previous similar messages [ 3321.161286] Lustre: server umount lustre-OST0000 complete [ 3332.064560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3332.067816] Lustre: Skipped 10 previous similar messages [ 3337.183347] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3337.275346] Lustre: server umount lustre-MDT0000 complete [ 3338.546711] LustreError: 212616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698757 with bad export cookie 6849703543559608665 [ 3338.551070] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3338.551437] LustreError: 212616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3338.713417] Lustre: server umount lustre-MDT0001 complete [ 3429.756591] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 3431.090806] Key type lgssc unregistered [ 3431.232437] LNet: 213679:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3431.234405] LNetError: 213679:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3431.241404] LNet: Removed LNI 192.168.203.159@tcp [ 3431.551092] Key type .llcrypt unregistered [ 3431.552070] Key type ._llcrypt unregistered [ 3438.995359] Key type ._llcrypt registered [ 3438.996331] Key type .llcrypt registered [ 3439.027898] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 3445.041323] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 3445.334069] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3445.357151] alg: No test for adler32 (adler32-zlib) [ 3446.211604] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 3446.294605] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 3447.871115] Key type lgssc registered [ 3448.185476] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3451.022983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3453.118915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3455.062193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3457.052301] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3459.757905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3459.791906] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3459.801733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3460.887974] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3460.899132] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3460.932822] Lustre: lustre-MDT0000: new disk, initializing [ 3460.956125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3460.962084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3462.134087] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3465.995752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3466.022169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3466.052034] Lustre: 216966:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3466.078707] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3466.081575] Lustre: Skipped 1 previous similar message [ 3466.117502] Lustre: lustre-MDT0001: new disk, initializing [ 3466.155957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3466.170260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3466.176626] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3467.390134] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3470.226686] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3471.245671] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3473.297424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3473.320447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3473.409387] Lustre: lustre-OST0000: new disk, initializing [ 3473.411271] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3473.437567] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3474.997324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3475.000441] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3475.034421] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3475.110445] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3477.962554] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3479.144730] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3479.209944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3480.384135] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3480.446928] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3485.154319] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3485.156249] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3485.156506] LustreError: Skipped 1 previous similar message [ 3485.160926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3485.162901] Lustre: Skipped 1 previous similar message [ 3487.084080] LustreError: 219880:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3487.145834] Lustre: server umount lustre-OST0000 complete [ 3490.272765] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3490.273292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3490.275932] Lustre: Skipped 2 previous similar messages [ 3494.330065] LustreError: 220081:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3494.331839] LustreError: 220081:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3494.355242] Lustre: server umount lustre-MDT0000 complete [ 3495.470606] LustreError: 216958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698914 with bad export cookie 6377436901723422840 [ 3495.475449] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3495.479997] LustreError: 216958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3495.549322] LustreError: 220282:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3495.552346] LustreError: 220282:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3495.633493] Lustre: server umount lustre-MDT0001 complete [ 3503.354409] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 3507.414701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3507.657213] LustreError: 221451:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3507.668247] LustreError: 221451:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 3507.683918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3509.178233] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3512.801196] LustreError: 221451:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3512.941763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3513.158269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3514.793582] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3515.715655] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3517.767836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3519.605408] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3522.164329] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3522.186508] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3522.234629] Lustre: lustre-OST0001: new disk, initializing [ 3522.236341] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3522.264131] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3522.265832] Lustre: Skipped 1 previous similar message [ 3523.966500] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3524.932995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3524.937205] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3524.974130] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3529.145778] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3532.595516] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3541.126150] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 10:09:20 (1755698960) [ 3543.521563] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3543.523944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3543.524795] Lustre: Skipped 3 previous similar messages [ 3543.526317] Lustre: Skipped 3 previous similar messages [ 3547.840833] LustreError: 224804:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3547.842723] LustreError: 224804:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3547.876883] Lustre: server umount lustre-MDT0000 complete [ 3548.642750] LustreError: 221451:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3548.646869] LustreError: 221451:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 3549.080451] LustreError: 224443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698967 with bad export cookie 6377436901723423708 [ 3549.083291] LustreError: 224443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3549.084960] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3549.237787] Lustre: server umount lustre-MDT0001 complete [ 3560.945131] LustreError: 225207:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3560.947307] LustreError: 225207:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 3560.962285] Lustre: server umount lustre-OST0000 complete [ 3571.250699] LustreError: 225408:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3571.252326] LustreError: 225408:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3571.331016] Lustre: server umount lustre-OST0001 complete [ 3573.291250] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 3576.219175] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 3579.392732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3582.723225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3585.891858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3589.233926] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3592.583511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3592.607894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3592.687560] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3592.698969] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3592.732714] Lustre: lustre-MDT0000: new disk, initializing [ 3592.756537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3592.762644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3594.245479] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3599.424217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3599.449122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3599.490321] Lustre: 227583:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3599.620155] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3600.910897] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3603.984610] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3605.088527] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3607.291375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3607.319201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3607.416697] Lustre: lustre-OST0000: new disk, initializing [ 3607.418745] Lustre: Skipped 1 previous similar message [ 3607.421825] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3607.425779] Lustre: Skipped 2 previous similar messages [ 3609.150812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3609.155142] Lustre: Skipped 1 previous similar message [ 3609.157206] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3609.196564] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3609.646945] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3612.636885] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3613.963690] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3614.043212] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3615.493533] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3615.578221] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3619.300831] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3619.301381] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3619.304603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3619.309826] Lustre: Skipped 1 previous similar message [ 3619.312615] Lustre: Skipped 1 previous similar message [ 3622.317219] LustreError: 230492:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3622.319188] LustreError: 230492:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3622.406981] Lustre: server umount lustre-OST0000 complete [ 3624.418324] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3624.419489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3624.426453] Lustre: Skipped 1 previous similar message [ 3630.461817] Lustre: server umount lustre-MDT0000 complete [ 3632.329263] LustreError: 229315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755699051 with bad export cookie 6377436901723425808 [ 3632.337393] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3632.338671] LustreError: 229315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3632.561781] Lustre: server umount lustre-MDT0001 complete [ 3636.575594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3636.845446] LustreError: 231370:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3636.884569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3636.888560] Lustre: Skipped 2 previous similar messages [ 3638.622851] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3642.336463] LustreError: 231370:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3642.439986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3644.307704] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3646.948963] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3648.187669] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3651.457432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3653.728823] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3656.260689] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3801.569555] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3801.574788] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3801.585174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3801.589332] Lustre: Skipped 3 previous similar messages [ 3805.672044] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3807.343174] LustreError: 233642:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3807.347081] LustreError: 233642:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3807.443526] Lustre: server umount lustre-OST0000 complete [ 3815.249589] Lustre: server umount lustre-MDT0000 complete [ 3815.904899] LustreError: 231370:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3815.913713] LustreError: 231370:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3816.791415] LustreError: 231349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755699235 with bad export cookie 6377436901723426690 [ 3816.798577] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3816.799117] LustreError: 231349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3816.987949] Lustre: server umount lustre-MDT0001 complete [ 3921.190905] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 3922.177106] Key type lgssc unregistered [ 3922.298445] LNet: 234607:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3922.300515] LNetError: 234607:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3922.309438] LNet: Removed LNI 192.168.203.159@tcp [ 3922.597094] Key type .llcrypt unregistered [ 3922.598170] Key type ._llcrypt unregistered [ 3931.159719] Key type ._llcrypt registered [ 3931.160769] Key type .llcrypt registered [ 3931.208280] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 3937.641549] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 3938.017374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3938.044045] alg: No test for adler32 (adler32-zlib) [ 3938.914524] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 3939.021641] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 3940.615175] Key type lgssc registered [ 3941.136145] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3944.490812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3946.720620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3949.117062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3951.452719] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3957.596006] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 3961.053572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3961.075945] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3961.080507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3962.163993] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3962.180264] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3962.237176] Lustre: lustre-MDT0000: new disk, initializing [ 3962.271303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3962.278719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3963.665460] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3968.383545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3968.407431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3968.459211] Lustre: 238586:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3968.493904] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3968.495674] Lustre: Skipped 1 previous similar message [ 3968.535828] Lustre: lustre-MDT0001: new disk, initializing [ 3968.582934] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3968.597754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3968.601105] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3969.981170] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3972.245846] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3974.616488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3974.642807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3974.756523] Lustre: lustre-OST0000: new disk, initializing [ 3974.758648] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3974.789350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3976.808156] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3979.267734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3979.272786] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3979.316293] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3980.954166] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3980.981170] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3981.062587] Lustre: lustre-OST0001: new disk, initializing [ 3981.066113] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3981.124682] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3983.666611] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3985.918264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3985.921293] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3985.953908] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3989.389547] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3994.568590] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4004.320060] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4004.323628] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4004.329599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4005.129650] LustreError: 242084:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4005.166599] Lustre: server umount lustre-MDT0000 complete [ 4006.368872] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4006.372428] Lustre: Skipped 1 previous similar message [ 4006.374041] LustreError: 238577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755699425 with bad export cookie 14325558975469445507 [ 4006.378308] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4006.378576] LustreError: 238577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4008.446350] LustreError: 242286:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 4008.448056] LustreError: 242286:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4008.523869] Lustre: server umount lustre-MDT0001 complete [ 4019.707141] LustreError: 242487:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4019.711231] LustreError: 242487:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4019.740065] Lustre: server umount lustre-OST0000 complete [ 4031.154368] LustreError: 242689:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4031.156533] LustreError: 242689:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4031.239728] Lustre: server umount lustre-OST0001 complete [ 4035.527921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4038.143933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4038.456717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4041.055460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4043.810749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4044.165698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4050.373123] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 4054.409972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4054.419125] Lustre: lustre-MDT0000: reset Object Index mappings [ 4054.612245] LustreError: 245990:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4054.616865] LustreError: 245990:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 4054.643478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4055.862191] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4058.878194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4058.885035] Lustre: lustre-MDT0001: reset Object Index mappings [ 4060.377085] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4061.491435] Lustre: 247088:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4063.810156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4063.929416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4063.931212] Lustre: Skipped 1 previous similar message [ 4064.932247] LustreError: 247600:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4064.934693] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4065.861168] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4069.030709] LustreError: 247441:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4069.039605] LustreError: 247441:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 4069.434506] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4071.410197] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 4071.567244] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4074.887113] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4076.005497] Lustre: 248920:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4084.704612] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4084.706504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4084.709310] Lustre: Skipped 1 previous similar message [ 4084.710890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4088.187465] LustreError: 249025:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4088.189174] LustreError: 249025:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4088.220481] Lustre: server umount lustre-MDT0000 complete [ 4089.586519] LustreError: 248923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755699508 with bad export cookie 14325558975469452283 [ 4089.593147] LustreError: 248923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4089.595499] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4089.781953] Lustre: server umount lustre-MDT0001 complete [ 4100.275123] LustreError: 249426:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 4100.276797] LustreError: 249426:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 4100.290793] Lustre: server umount lustre-OST0000 complete [ 4111.930379] Lustre: server umount lustre-OST0001 complete [ 4117.751644] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 10:18:56 (1755699536) [ 4122.574472] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 4125.343599] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 4128.613503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4130.637550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4133.287478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4135.699234] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4138.482394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4138.519166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4138.642134] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4138.660652] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4138.708637] Lustre: lustre-MDT0000: new disk, initializing [ 4138.738477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4138.740405] Lustre: Skipped 1 previous similar message [ 4138.746179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4140.252789] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4144.108685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4144.129895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4144.158479] Lustre: 253131:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4144.254367] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4145.521341] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4148.430175] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4149.475756] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4151.595901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4151.621723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4151.705966] Lustre: lustre-OST0000: new disk, initializing [ 4151.707264] Lustre: Skipped 1 previous similar message [ 4151.708773] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4151.710582] Lustre: Skipped 2 previous similar messages [ 4153.012893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4153.015181] Lustre: Skipped 1 previous similar message [ 4153.016621] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4153.048561] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4153.466513] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4156.326454] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4157.583904] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4157.657397] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4158.871975] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4158.947572] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4161.581864] LustreError: 256040:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4161.583714] LustreError: 256040:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4161.644741] Lustre: server umount lustre-OST0000 complete [ 4163.554131] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4163.554933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4163.557479] Lustre: Skipped 4 previous similar messages [ 4163.559267] Lustre: Skipped 4 previous similar messages [ 4168.672308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4168.674409] Lustre: Skipped 2 previous similar messages [ 4173.792641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4173.794436] Lustre: Skipped 1 previous similar message [ 4177.375133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4177.430127] Lustre: server umount lustre-MDT0000 complete [ 4178.588270] LustreError: 253122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755699597 with bad export cookie 14325558975469454376 [ 4178.591235] LustreError: 253122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4178.591533] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4189.280439] Lustre: server umount lustre-MDT0001 complete [ 4197.509425] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 4202.524798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4202.722981] LustreError: 257610:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4202.752434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4202.756466] Lustre: Skipped 2 previous similar messages [ 4204.484798] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4208.098175] LustreError: 257610:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4208.772655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4210.725779] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4211.679475] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4213.775584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4215.723160] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4218.444529] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4218.473993] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4218.538762] Lustre: lustre-OST0001: new disk, initializing [ 4218.540843] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4219.973882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4219.977189] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4220.019877] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4220.418563] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4225.396582] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4227.041793] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4234.605894] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 4234.638099] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 4238.818741] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4238.819763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4238.822613] Lustre: Skipped 3 previous similar messages [ 4238.824128] Lustre: Skipped 2 previous similar messages [ 4242.812952] LustreError: 261093:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4242.814878] LustreError: 261093:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4242.842831] Lustre: server umount lustre-MDT0000 complete [ 4243.936228] LustreError: 257609:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4243.940282] LustreError: 257609:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 4243.967032] LustreError: 261007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755699662 with bad export cookie 14325558975469455258 [ 4243.969360] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4243.970162] LustreError: 261007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4244.120843] Lustre: server umount lustre-MDT0001 complete [ 4256.261085] Lustre: server umount lustre-OST0000 complete [ 4267.789231] Lustre: server umount lustre-OST0001 complete [ 4274.605373] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 4278.362969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4278.496178] LustreError: 262865:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4278.513578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4278.516222] Lustre: Skipped 3 previous similar messages [ 4279.729211] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4282.649263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4284.187348] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4285.328455] Lustre: 263962:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4287.841745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4289.073142] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 4290.189699] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4293.312539] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4294.660964] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 4295.612380] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4298.845878] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4300.370093] Lustre: 265805:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4307.005329] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 4307.013174] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 4309.778713] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 10:22:08 (1755699728) [ 4313.568614] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4313.570748] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4313.574483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4313.575935] Lustre: Skipped 1 previous similar message [ 4316.797817] LustreError: 266490:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4316.799625] LustreError: 266490:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 4316.835994] Lustre: server umount lustre-MDT0000 complete [ 4317.996610] LustreError: 262845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755699736 with bad export cookie 14325558975469461530 [ 4318.001745] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4318.152302] Lustre: server umount lustre-MDT0001 complete [ 4329.607102] Lustre: server umount lustre-OST0000 complete [ 4341.122304] Lustre: server umount lustre-OST0001 complete [ 4347.063549] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 4349.842640] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 4352.981558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4355.749841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4358.310033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4360.679353] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4364.025098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4364.064347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4364.191898] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4364.211799] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4364.276645] Lustre: lustre-MDT0000: new disk, initializing [ 4364.316929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4365.836774] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4369.979877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4369.999638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4370.029150] Lustre: 269920:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4370.124863] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4371.316711] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4374.348687] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4375.468579] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4377.637414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4377.663276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4378.905031] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4379.498458] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4382.427029] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4383.120186] Lustre: Mounted lustre-client [ 4383.415311] LustreError: 272405:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff8f6286113800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 4383.433634] Lustre: Unmounted lustre-client [ 4388.324163] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4388.325454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4388.327115] Lustre: Skipped 4 previous similar messages [ 4388.328688] Lustre: Skipped 4 previous similar messages [ 4392.374046] Lustre: server umount lustre-OST0000 complete [ 4395.999903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4399.636577] Lustre: server umount lustre-MDT0000 complete [ 4400.794149] LustreError: 271654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755699819 with bad export cookie 14325558975469468908 [ 4400.797020] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4400.797063] LustreError: 271654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4400.941904] Lustre: server umount lustre-MDT0001 complete [ 4404.140149] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 4405.179662] Key type lgssc unregistered [ 4442.207349] LNet: 273676:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4442.209549] LNetError: 273676:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4514.917585] LNet: Removed LNI 192.168.203.159@tcp [ 4515.374099] Key type .llcrypt unregistered [ 4515.375080] Key type ._llcrypt unregistered [ 4522.433375] Key type ._llcrypt registered [ 4522.434471] Key type .llcrypt registered [ 4522.465630] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 4529.789440] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 4530.210276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4530.226220] alg: No test for adler32 (adler32-zlib) [ 4531.064266] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 4531.146823] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 4532.719176] Key type lgssc registered [ 4533.323740] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4536.771701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4539.838417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4542.296516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4544.625510] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4545.281676] Lustre: DEBUG MARKER: == conf-sanity test 200a: check CPU partitions =========== 10:26:04 (1755699964) [ 4565.513413] Lustre: DEBUG MARKER: == conf-sanity test 200b: set CPU pattern using core selection ========================================================== 10:26:24 (1755699984) [ 4580.417567] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 4581.362571] Key type lgssc unregistered [ 4581.486377] LNet: 279374:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4581.490153] LNetError: 279374:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4581.498293] LNet: Removed LNI 192.168.203.159@tcp [ 4581.764415] Key type .llcrypt unregistered [ 4581.765385] Key type ._llcrypt unregistered [ 4606.595177] Key type ._llcrypt registered [ 4606.596149] Key type .llcrypt registered [ 4606.629327] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 4606.726270] Key type .llcrypt unregistered [ 4606.727337] Key type ._llcrypt unregistered [ 4616.780489] Key type ._llcrypt registered [ 4616.781389] Key type .llcrypt registered [ 4616.813840] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 4616.910161] Key type .llcrypt unregistered [ 4616.911139] Key type ._llcrypt unregistered [ 4641.706385] Key type ._llcrypt registered [ 4641.707395] Key type .llcrypt registered [ 4641.748896] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 4641.856305] Key type .llcrypt unregistered [ 4641.857277] Key type ._llcrypt unregistered [ 4651.966354] Key type ._llcrypt registered [ 4651.967321] Key type .llcrypt registered [ 4652.003838] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 4652.106278] Key type .llcrypt unregistered [ 4652.107213] Key type ._llcrypt unregistered [ 4661.962449] Key type ._llcrypt registered [ 4661.963590] Key type .llcrypt registered [ 4661.998256] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 4662.090315] Key type .llcrypt unregistered [ 4662.091277] Key type ._llcrypt unregistered [ 4686.757661] Key type ._llcrypt registered [ 4686.758670] Key type .llcrypt registered [ 4686.794431] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 4686.895327] Key type .llcrypt unregistered [ 4686.896579] Key type ._llcrypt unregistered [ 4696.885859] Key type ._llcrypt registered [ 4696.886744] Key type .llcrypt registered [ 4696.920913] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 4697.015261] Key type .llcrypt unregistered [ 4697.016101] Key type ._llcrypt unregistered [ 4706.916255] Key type ._llcrypt registered [ 4706.917194] Key type .llcrypt registered [ 4706.954328] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 4707.060363] Key type .llcrypt unregistered [ 4707.061286] Key type ._llcrypt unregistered [ 4723.781577] Key type ._llcrypt registered [ 4723.782566] Key type .llcrypt registered [ 4723.813725] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 4729.438262] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 4729.838801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4729.849238] alg: No test for adler32 (adler32-zlib) [ 4730.702404] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 4730.789611] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 4732.367121] Key type lgssc registered [ 4732.689903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4735.719424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4738.161980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4740.044167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4742.201637] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4742.772575] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 4631 sec ========== 10:29:21 (1755700161) [ 4743.360271] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 10:29:22 (1755700162) === [ 4744.440333] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 10:29:23 (1755700163) === [ 4756.923356] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 4757.922453] Key type lgssc unregistered [ 4758.047308] LNet: 295171:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4758.049367] LNetError: 295171:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4758.056263] LNet: Removed LNI 192.168.203.159@tcp [ 4758.317376] Key type .llcrypt unregistered [ 4758.318383] Key type ._llcrypt unregistered