[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-1.fc39 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 540179688 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002351] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004018] kvm-guest: setup PV IPIs [ 0.006571] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008004] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010131] LSM: Security Framework initializing [ 0.011052] Yama: becoming mindful. [ 0.012030] SELinux: Initializing. [ 0.013064] *** VALIDATE selinux *** [ 0.021704] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026466] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027140] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028101] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029109] *** VALIDATE tmpfs *** [ 0.031150] *** VALIDATE proc *** [ 0.032202] *** VALIDATE cgroup *** [ 0.033008] *** VALIDATE cgroup2 *** [ 0.035063] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037119] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039024] Spectre V2 : User space: Vulnerable [ 0.040008] Speculative Store Bypass: Vulnerable [ 0.043100] debug: unmapping init [mem 0xffffffff94059000-0xffffffff94060fff] [ 0.045178] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046585] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047023] ... version: 2 [ 0.048010] ... bit width: 48 [ 0.049010] ... generic registers: 4 [ 0.050009] ... value mask: 0000ffffffffffff [ 0.051013] ... max period: 00007fffffffffff [ 0.052011] ... fixed-purpose events: 3 [ 0.053010] ... event mask: 000000070000000f [ 0.054268] rcu: Hierarchical SRCU implementation. [ 0.056369] smp: Bringing up secondary CPUs ... [ 0.057523] x86: Booting SMP configuration: [ 0.058020] .... node #0, CPUs: #1 #2 #3 [ 0.073510] smp: Brought up 1 node, 4 CPUs [ 0.075035] smpboot: Max logical packages: 1 [ 0.076010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.233409] node 0 deferred pages initialised in 154ms [ 0.236009] devtmpfs: initialized [ 0.237074] x86/mm: Memory block size: 128MB [ 0.239493] gcov: version magic: 0x41383552 [ 0.242213] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.243063] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.244233] pinctrl core: initialized pinctrl subsystem [ 0.245175] [ 0.245552] ************************************************************* [ 0.246013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.247011] ** ** [ 0.248010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.249011] ** ** [ 0.250010] ** This means that this kernel is built to expose internal ** [ 0.251017] ** IOMMU data structures, which may compromise security on ** [ 0.252011] ** your system. ** [ 0.253008] ** ** [ 0.254010] ** If you see this message and you are not debugging the ** [ 0.255012] ** kernel, report this immediately to your vendor! ** [ 0.256012] ** ** [ 0.257010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.258012] ************************************************************* [ 0.259592] NET: Registered protocol family 16 [ 0.260354] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.261059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.262050] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.265054] cpuidle: using governor menu [ 0.267795] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.270561] PCI: Using configuration type 1 for base access [ 0.272140] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.281147] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.283031] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.287057] cryptd: max_cpu_qlen set to 1000 [ 0.292325] ACPI: Added _OSI(Module Device) [ 0.294024] ACPI: Added _OSI(Processor Device) [ 0.295016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.296011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.300644] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.309666] ACPI: Interpreter enabled [ 0.310053] ACPI: PM: (supports S0 S3 S4 S5) [ 0.311010] ACPI: Using IOAPIC for interrupt routing [ 0.313113] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.315424] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.326060] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.328042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.330018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.333090] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.337381] acpiphp: Slot [2] registered [ 0.338132] acpiphp: Slot [5] registered [ 0.340115] acpiphp: Slot [6] registered [ 0.341082] acpiphp: Slot [7] registered [ 0.342074] acpiphp: Slot [8] registered [ 0.343081] acpiphp: Slot [9] registered [ 0.344146] acpiphp: Slot [10] registered [ 0.345077] acpiphp: Slot [3] registered [ 0.347063] acpiphp: Slot [4] registered [ 0.348078] acpiphp: Slot [11] registered [ 0.349077] acpiphp: Slot [12] registered [ 0.351069] acpiphp: Slot [13] registered [ 0.352099] acpiphp: Slot [14] registered [ 0.354108] acpiphp: Slot [15] registered [ 0.355079] acpiphp: Slot [16] registered [ 0.356074] acpiphp: Slot [17] registered [ 0.357063] acpiphp: Slot [18] registered [ 0.358059] acpiphp: Slot [19] registered [ 0.359062] acpiphp: Slot [20] registered [ 0.361099] acpiphp: Slot [21] registered [ 0.362062] acpiphp: Slot [22] registered [ 0.363068] acpiphp: Slot [23] registered [ 0.364095] acpiphp: Slot [24] registered [ 0.365071] acpiphp: Slot [25] registered [ 0.366072] acpiphp: Slot [26] registered [ 0.368091] acpiphp: Slot [27] registered [ 0.369065] acpiphp: Slot [28] registered [ 0.370096] acpiphp: Slot [29] registered [ 0.371072] acpiphp: Slot [30] registered [ 0.373072] acpiphp: Slot [31] registered [ 0.374072] PCI host bridge to bus 0000:00 [ 0.375015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.377018] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.379021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.381017] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.383019] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.385028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.387211] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.390010] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.392442] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.402895] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.407061] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.409020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.413033] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.415016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.418544] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.420661] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.422034] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.424817] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.431013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.444031] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.448014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.454266] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.460018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.470016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.484020] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.493044] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.502020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.511025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.528021] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.539957] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.545020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.550016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.562020] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.569000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.575015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.579020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.592026] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.600988] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.611018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.616017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.629027] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.639071] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.643016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.649016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.660019] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.674000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.676375] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.678382] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.680407] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.683247] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.689052] iommu: Default domain type: Passthrough [ 0.690398] SCSI subsystem initialized [ 0.691114] ACPI: bus type USB registered [ 0.692137] usbcore: registered new interface driver usbfs [ 0.693101] usbcore: registered new interface driver hub [ 0.695115] usbcore: registered new device driver usb [ 0.696161] pps_core: LinuxPPS API ver. 1 registered [ 0.697008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.700091] PTP clock support registered [ 0.702076] EDAC MC: Ver: 3.0.0 [ 0.703435] PCI: Using ACPI for IRQ routing [ 0.704813] NetLabel: Initializing [ 0.706013] NetLabel: domain hash size = 128 [ 0.708013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.709084] NetLabel: unlabeled traffic allowed by default [ 0.711156] vgaarb: loaded [ 0.713311] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.715018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.721395] clocksource: Switched to clocksource kvm-clock [ 0.826043] VFS: Disk quotas dquot_6.6.0 [ 0.827427] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.829717] *** VALIDATE ramfs *** [ 0.830958] *** VALIDATE hugetlbfs *** [ 0.832465] pnp: PnP ACPI init [ 0.834552] pnp: PnP ACPI: found 6 devices [ 0.851837] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.854908] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.856984] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.859092] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.861365] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.863573] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 0.866255] NET: Registered protocol family 2 [ 0.868341] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.872918] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.876444] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.881706] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.884946] TCP: Hash tables configured (established 65536 bind 65536) [ 0.887789] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.890671] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.893355] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.895975] NET: Registered protocol family 1 [ 0.897972] RPC: Registered named UNIX socket transport module. [ 0.899499] RPC: Registered udp transport module. [ 0.900667] RPC: Registered tcp transport module. [ 0.902320] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.904347] NET: Registered protocol family 44 [ 0.906351] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.908208] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.910080] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.912196] PCI: CLS 0 bytes, default 64 [ 0.913585] Unpacking initramfs... [ 2.401563] debug: unmapping init [mem 0xffff932afcc54000-0xffff932afffbffff] [ 2.408576] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.411227] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.416250] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.915629] Initialise system trusted keyrings [ 2.917194] Key type blacklist registered [ 2.918871] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.927476] zbud: loaded [ 2.930031] *** VALIDATE nfs *** [ 2.931165] *** VALIDATE nfs4 *** [ 2.932543] pstore: using deflate compression [ 2.935649] Platform Keyring initialized [ 3.038817] NET: Registered protocol family 38 [ 3.041483] Key type asymmetric registered [ 3.042893] Asymmetric key parser 'x509' registered [ 3.044660] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.047220] io scheduler mq-deadline registered [ 3.048882] io scheduler kyber registered [ 3.050433] io scheduler bfq registered [ 3.051425] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.053769] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.055870] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.058121] ACPI: Power Button [PWRF] [ 3.153575] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.250567] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.449761] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.541514] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.734914] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.768900] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.800041] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.804404] Non-volatile memory driver v1.3 [ 3.806084] Linux agpgart interface v0.103 [ 3.841244] virtio_blk virtio1: [vda] 133800 512-byte logical blocks (68.5 MB/65.3 MiB) [ 3.844416] vda: detected capacity change from 0 to 68505600 [ 3.861967] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.864656] vdb: detected capacity change from 0 to 1073741824 [ 3.882054] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.884519] vdc: detected capacity change from 0 to 2621440000 [ 3.897438] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.900495] vdd: detected capacity change from 0 to 2621440000 [ 3.919724] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.922607] vde: detected capacity change from 0 to 4294967296 [ 3.937922] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.940871] vdf: detected capacity change from 0 to 4294967296 [ 3.947607] libphy: Fixed MDIO Bus: probed [ 3.955744] usbcore: registered new interface driver usbserial_generic [ 3.959337] usbserial: USB Serial support registered for generic [ 3.961232] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.965096] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.966605] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.968756] mousedev: PS/2 mouse device common for all mice [ 3.970827] rtc_cmos 00:05: RTC can wake from S4 [ 3.973778] rtc_cmos 00:05: registered as rtc0 [ 3.974016] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.975402] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.980393] intel_pstate: CPU model not supported [ 3.983051] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.986442] hid: raw HID events driver (C) Jiri Kosina [ 3.988535] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.988590] usbcore: registered new interface driver usbhid [ 3.993132] usbhid: USB HID core driver [ 3.994538] drop_monitor: Initializing network drop monitor service [ 3.997286] Initializing XFRM netlink socket [ 3.999446] NET: Registered protocol family 10 [ 4.002445] Segment Routing with IPv6 [ 4.003972] NET: Registered protocol family 17 [ 4.005596] mpls_gso: MPLS GSO support [ 4.011347] RAS: Correctable Errors collector initialized. [ 4.013209] AVX version of gcm_enc/dec engaged. [ 4.014533] AES CTR mode by8 optimization enabled [ 4.107984] sched_clock: Marking stable (4107963599, 0)->(5071190588, -963226989) [ 4.111303] registered taskstats version 1 [ 4.113681] Loading compiled-in X.509 certificates [ 4.115555] zswap: loaded using pool lzo/zbud [ 4.142427] Key type big_key registered [ 4.155822] Key type encrypted registered [ 4.158064] ima: No TPM chip found, activating TPM-bypass! [ 4.159932] ima: Allocated hash algorithm: sha1 [ 4.162517] ima: No architecture policies found [ 4.164476] evm: Initialising EVM extended attributes: [ 4.166316] evm: security.selinux [ 4.167506] evm: security.ima [ 4.168725] evm: security.capability [ 4.170049] evm: HMAC attrs: 0x1 [ 4.172799] rtc_cmos 00:05: setting system clock to 2025-10-23 16:37:20 UTC (1761237440) [ 4.179439] debug: unmapping init [mem 0xffffffff95003000-0xffffffff951fffff] [ 4.182670] debug: unmapping init [mem 0xffffffff93d82000-0xffffffff94058fff] [ 4.195072] Write protecting the kernel read-only data: 28672k [ 4.199568] debug: unmapping init [mem 0xffffffff92403000-0xffffffff925fffff] [ 4.202984] debug: unmapping init [mem 0xffffffff92d14000-0xffffffff92dfffff] [ 4.239287] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.248251] systemd[1]: Detected virtualization kvm. [ 4.252341] systemd[1]: Detected architecture x86-64. [ 4.254209] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.283913] systemd[1]: No hostname configured. [ 4.285363] systemd[1]: Set hostname to . [ 4.287474] random: systemd: uninitialized urandom read (16 bytes read) [ 4.289468] systemd[1]: Initializing machine ID from random generator. [ 4.420654] random: systemd: uninitialized urandom read (16 bytes read) [ 4.423457] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.431948] random: systemd: uninitialized urandom read (16 bytes read) [ 4.433603] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.444153] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.078615] device-mapper: uevent: version 1.0.3 [ 5.081471] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.797230] virtio_net virtio0 ens2: renamed from eth0 [ 5.811368] random: fast init done [ 5.831393] scsi host0: ata_piix [ 5.853451] scsi host1: ata_piix [ 5.855330] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.857616] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.529396] dracut-initqueue[583]: RTNETLINK answers: File exists [ 10.680296] random: crng init done [ 10.682461] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.125474] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.274583] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.521956] SELinux: Disabled at runtime. [ 12.577370] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.586503] systemd[1]: Detected virtualization kvm. [ 12.591387] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.118305] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.120917] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.130116] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.134654] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.137649] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.151670] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.163290] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ 13.364917] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 13.647228] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.952820] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.953479] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.098300] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.117212] EDAC sbridge: Ver: 1.1.2 [ 15.840210] Key type dns_resolver registered [ 16.148779] NFS: Registering the id_resolver key type [ 16.150423] Key type id_resolver registered [ 16.151399] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg620-server login: [ 68.157724] libcfs: loading out-of-tree module taints kernel. [ 68.189741] Key type ._llcrypt registered [ 68.193464] Key type .llcrypt registered [ 68.273426] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [ 83.473095] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 84.587796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 84.607472] alg: No test for adler32 (adler32-zlib) [ 86.172573] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 87.665961] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 89.537866] Key type lgssc registered [ 91.624789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 93.001032] hrtimer: interrupt took 3384545 ns [ 115.084220] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 118.312706] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 128.450271] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 139.160514] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 152.390081] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 176.768063] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 193.822084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 194.003497] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 194.061274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 195.566595] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 195.642867] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 195.896860] Lustre: lustre-MDT0000: new disk, initializing [ 196.098975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 196.142551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 201.691723] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 215.831305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 215.954138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 216.010796] Lustre: 6546:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 216.048385] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 216.050552] Lustre: Skipped 1 previous similar message [ 216.111463] Lustre: lustre-MDT0001: new disk, initializing [ 216.168820] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 216.190092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 216.201843] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 221.132563] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 226.734584] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 239.078271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 239.157043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 239.539979] Lustre: lustre-OST0000: new disk, initializing [ 239.559525] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 239.673597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 240.398155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 240.403439] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 240.485795] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 245.064121] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 258.627099] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 258.710793] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 258.795645] Lustre: lustre-OST0001: new disk, initializing [ 258.802803] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 258.867649] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 264.637895] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 264.755023] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 264.775647] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 264.830366] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 275.436235] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 281.748348] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 291.634565] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing check_logdir /tmp/testlogs/ [ 295.405879] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing yml_node [ 299.114233] Lustre: DEBUG MARKER: Client: 2.16.59.38 [ 301.432907] Lustre: DEBUG MARKER: MDS: 2.16.59.38 [ 303.527456] Lustre: DEBUG MARKER: OSS: 2.16.59.38 [ 304.980585] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Oct 23 12:42:19 EDT 2025 [ 320.363017] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 321.906229] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 336.356867] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 336.357462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 336.376136] Lustre: Skipped 3 previous similar messages [ 336.382248] Lustre: Skipped 3 previous similar messages [ 341.472660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 341.475735] Lustre: Skipped 3 previous similar messages [ 342.572834] LustreError: 12255:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 342.735976] Lustre: server umount lustre-MDT0000 complete [ 345.877946] LustreError: 6538:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761237782 with bad export cookie 1300324321026645210 [ 345.883307] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 345.889811] LustreError: 6538:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 346.598264] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 346.602416] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 346.607695] Lustre: Skipped 1 previous similar message [ 350.624076] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 350.627576] Lustre: Skipped 2 previous similar messages [ 352.035238] LustreError: 12457:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 352.041920] LustreError: 12457:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 352.259456] Lustre: server umount lustre-MDT0001 complete [ 355.506528] LustreError: 12658:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 355.523266] LustreError: 12658:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 355.604480] Lustre: server umount lustre-OST0000 complete [ 358.781584] LustreError: 12859:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 358.792039] LustreError: 12859:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 358.954662] Lustre: server umount lustre-OST0001 complete [ 364.304963] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [ 372.555857] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 380.650632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 385.556396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 390.922333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 396.556233] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 402.346518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 402.400264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 402.634629] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 402.654615] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 402.751661] Lustre: lustre-MDT0000: new disk, initializing [ 402.874571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 402.906907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 406.268997] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 414.770268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 414.825206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 414.903619] Lustre: 15045:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 414.984889] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 414.988869] Lustre: Skipped 1 previous similar message [ 415.051978] Lustre: lustre-MDT0001: new disk, initializing [ 415.154411] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 415.197698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 415.206238] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 415.347868] Lustre: 15841:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 415.370726] Lustre: 15841:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 415.376339] Lustre: 15841:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 415.379901] Lustre: 15841:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 415.386946] Lustre: 15841:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 415.391837] Lustre: 15841:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 418.490268] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 425.019956] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 427.978481] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 434.012873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 434.073618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 434.263562] Lustre: lustre-OST0000: new disk, initializing [ 434.266120] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 434.338208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 435.571421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 435.588029] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 435.663522] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 438.312066] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 444.642790] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 448.132527] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 448.323483] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 451.669144] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 451.900147] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 456.164745] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 456.168174] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 456.169187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 456.184601] Lustre: Skipped 1 previous similar message [ 459.589617] LustreError: 17983:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 459.603917] LustreError: 17983:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 459.764400] Lustre: server umount lustre-OST0000 complete [ 466.405386] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 466.408900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 466.419127] Lustre: Skipped 1 previous similar message [ 466.421990] Lustre: Skipped 1 previous similar message [ 466.430633] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 466.437384] LustreError: Skipped 1 previous similar message [ 477.151121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 477.250174] LustreError: 18184:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 477.253621] LustreError: 18184:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 477.351013] Lustre: server umount lustre-MDT0000 complete [ 480.481936] LustreError: 15036:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761237916 with bad export cookie 1300324321026646785 [ 480.491733] LustreError: 15036:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 480.492433] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 480.827857] Lustre: server umount lustre-MDT0001 complete [ 486.079047] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 12:45:20 (1761237920) [ 501.538294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 501.880251] LustreError: 19524:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 501.890942] LustreError: 19524:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 501.940313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 505.589492] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 507.362319] LustreError: 19524:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 512.485209] LustreError: 19524:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 513.071070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 516.984531] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 521.138950] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 524.241259] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 526.001656] Lustre: Failing over lustre-MDT0000 [ 526.129533] LustreError: 20867:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 526.137581] LustreError: 20867:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 526.193548] Lustre: server umount lustre-MDT0000 complete [ 527.847715] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 527.854925] LustreError: 20236:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 527.866693] Lustre: Skipped 1 previous similar message [ 532.961607] LustreError: 20236:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 532.986256] LustreError: 20236:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 543.200539] LustreError: 19523:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 543.223768] LustreError: 19523:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 544.031095] Lustre: 3662:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761237964/real 1761237964] req@ffff932a426b5880 x1846791397074048/t0(0) o400->MGC192.168.206.120@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761237980 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 544.059701] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 544.195690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 554.465506] LustreError: 3659:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff932b4468e300 x1846791397076864/t0(0) o250->MGC192.168.206.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 557.731482] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 557.750926] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 557.773768] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 558.618734] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 566.202092] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 573.409936] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 573.417405] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 573.450080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 573.452561] Lustre: Skipped 4 previous similar messages [ 576.358841] LustreError: 21848:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 576.362017] LustreError: 21848:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 576.434782] Lustre: server umount lustre-MDT0000 complete [ 578.870814] LustreError: 19503:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761238015 with bad export cookie 1300324321026648241 [ 578.874655] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 578.883055] LustreError: 19503:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 579.114498] Lustre: server umount lustre-MDT0001 complete [ 585.233867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 585.565691] LustreError: 22529:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 585.587369] LustreError: 22529:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 585.632665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 585.636714] Lustre: Skipped 2 previous similar messages [ 588.461969] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 594.306257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 597.286888] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 601.300714] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 604.238583] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 606.124471] Lustre: Failing over lustre-MDT0000 [ 606.290479] Lustre: server umount lustre-MDT0000 complete [ 609.256477] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 609.267946] Lustre: Skipped 1 previous similar message [ 619.489275] LustreError: 22528:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 619.495982] LustreError: 22528:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 623.115960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 623.189152] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 623.450000] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 626.286387] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 628.707063] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 628.715912] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 628.729474] Lustre: Skipped 1 previous similar message [ 628.761943] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 632.667354] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 637.925954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 637.928243] Lustre: Skipped 2 previous similar messages [ 650.207121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 650.321175] LustreError: 24853:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 650.329317] LustreError: 24853:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 650.418715] Lustre: server umount lustre-MDT0000 complete [ 653.375138] LustreError: 22509:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761238089 with bad export cookie 1300324321026649172 [ 653.388576] LustreError: 22509:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 653.391702] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 653.761822] Lustre: server umount lustre-MDT0001 complete [ 660.477722] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 12:48:15 (1761238095) [ 672.629671] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [ 681.223834] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 690.116904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 695.927662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 701.502741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 707.209325] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 715.519976] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 724.159729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 724.200490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 724.343587] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 724.377361] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 724.422475] Lustre: lustre-MDT0000: new disk, initializing [ 724.531083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 724.534602] Lustre: Skipped 2 previous similar messages [ 724.548275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 727.790544] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 737.245825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 737.303439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 737.393113] Lustre: 28528:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 737.633860] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 737.729885] Lustre: 29323:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 737.739334] Lustre: 29323:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 1 previous similar message [ 737.751980] Lustre: 29323:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 737.768631] Lustre: 29323:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 1 previous similar message [ 737.782732] Lustre: 29323:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 737.796656] Lustre: 29323:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 737.823828] Lustre: 29323:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 737.844104] Lustre: 29323:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 737.855218] Lustre: 29323:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 737.869249] Lustre: 29323:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 737.884474] Lustre: 29323:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 737.898604] Lustre: 29323:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 741.222613] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 748.196101] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 751.980270] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 758.012696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 758.112749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 759.951358] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 759.994927] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 760.028626] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 760.039645] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 763.118836] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 769.300469] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 770.075179] Lustre: 28535:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 511, rollback = 2 [ 770.086616] Lustre: 28535:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 1 previous similar message [ 770.090319] Lustre: 28535:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 2/8/6, destroy: 0/0/0 [ 770.099203] Lustre: 28535:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 1 previous similar message [ 770.108142] Lustre: 28535:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 5/511/0 [ 770.116790] Lustre: 28535:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 770.121606] Lustre: 28535:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 5/58/0, punch: 0/0/0, quota 1/3/0 [ 770.125702] Lustre: 28535:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 770.139852] Lustre: 28535:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 5/102/3, delete: 0/0/0 [ 770.144927] Lustre: 28535:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 770.149507] Lustre: 28535:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 770.152773] Lustre: 28535:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 775.622543] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 791.009205] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 791.009319] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 791.012668] LustreError: Skipped 1 previous similar message [ 791.018274] Lustre: Skipped 3 previous similar messages [ 791.023095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 791.030079] Lustre: Skipped 7 previous similar messages [ 795.691242] LustreError: 31187:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 795.695502] LustreError: 31187:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 795.956437] Lustre: server umount lustre-OST0000 complete [ 805.890789] Lustre: server umount lustre-MDT0000 complete [ 806.370052] LustreError: 28538:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 806.395436] LustreError: 28538:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 808.990397] LustreError: 30274:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761238245 with bad export cookie 1300324321026649564 [ 808.993895] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 809.001116] LustreError: 30274:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 809.449010] Lustre: server umount lustre-MDT0001 complete [ 819.512881] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 822.240920] Key type lgssc unregistered [ 822.563676] LNet: 32121:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 822.573407] LNetError: 32121:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 822.592168] LNet: Removed LNI 192.168.206.120@tcp [ 823.417139] Key type .llcrypt unregistered [ 823.419380] Key type ._llcrypt unregistered [ 844.346120] Key type ._llcrypt registered [ 844.356133] Key type .llcrypt registered [ 844.468062] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 845.293931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 845.383097] alg: No test for adler32 (adler32-zlib) [ 846.370140] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 846.539319] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 848.175183] Key type lgssc registered [ 849.084304] Lustre: Echo OBD driver; http://www.lustre.org/ [ 855.924689] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 855.943966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 857.346217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 860.702103] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 866.798532] LustreError: 33517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 866.807680] LustreError: 33517:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 868.746234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 869.283984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 873.097574] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 877.699408] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 880.670831] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 886.465503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 886.883810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 892.271492] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 894.075134] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 897.126129] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 912.844225] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 12:52:27 (1761238347) [ 914.563675] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 916.356140] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 922.900518] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 12:52:37 (1761238357) [ 930.728331] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 12:52:45 (1761238365) [ 931.762458] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 939.126759] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 939.133183] Lustre: Skipped 2 previous similar messages [ 940.896502] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 12:52:55 (1761238375) [ 943.917296] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 943.920117] Lustre: Skipped 3 previous similar messages [ 958.505312] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 958.508811] Lustre: Skipped 6 previous similar messages [ 960.590179] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 12:53:15 (1761238395) [ 962.825316] Lustre: *** cfs_fail_loc=131b, val=2*** [ 965.845383] Lustre: *** cfs_fail_loc=131b, val=2*** [ 972.644906] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 12:53:27 (1761238407) [ 975.708382] Lustre: Setting parameter general.jobid_name=TESTNAME1 in log params [ 975.712102] Lustre: Skipped 5 previous similar messages [ 983.293700] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 12:53:38 (1761238418) [ 994.787332] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 994.791103] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 994.791584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 994.796183] Lustre: Skipped 1 previous similar message [ 998.007226] LustreError: 40074:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 998.257955] Lustre: server umount lustre-OST0000 complete [ 1002.467101] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1002.476409] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1002.496319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1002.499065] Lustre: Skipped 1 previous similar message [ 1005.024136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1005.025910] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1007.570548] LustreError: 40275:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1007.588051] LustreError: 40275:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1007.713977] Lustre: server umount lustre-MDT0000 complete [ 1010.144625] LustreError: 33518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1011.258446] LustreError: 33498:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761238447 with bad export cookie 14215565865403955092 [ 1011.276437] LustreError: 33498:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1011.277022] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1011.430918] LustreError: 40477:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1011.432771] LustreError: 40477:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1011.680645] Lustre: server umount lustre-MDT0001 complete [ 1020.429528] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 1022.892942] Key type lgssc unregistered [ 1023.172685] LNet: 41008:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1023.193033] LNetError: 41008:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1023.216129] LNet: Removed LNI 192.168.206.120@tcp [ 1023.990765] Key type .llcrypt unregistered [ 1023.993817] Key type ._llcrypt unregistered [ 1039.308955] Key type ._llcrypt registered [ 1039.310497] Key type .llcrypt registered [ 1039.411379] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [ 1053.138577] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 1054.083784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1054.180287] alg: No test for adler32 (adler32-zlib) [ 1055.260693] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 1055.554384] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 1057.233080] Key type lgssc registered [ 1058.336099] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1066.162534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1072.100637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1076.935135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1082.171797] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1088.033858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1088.111203] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1088.127718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1089.373445] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1089.404328] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1089.481782] Lustre: lustre-MDT0000: new disk, initializing [ 1089.588251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1089.602307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1092.810183] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 1101.699265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1101.783587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1101.848647] Lustre: 44134:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1101.897734] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1101.905177] Lustre: Skipped 1 previous similar message [ 1101.953868] Lustre: lustre-MDT0001: new disk, initializing [ 1102.039404] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1102.088441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1102.093455] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1102.206132] Lustre: 44930:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 1102.212957] Lustre: 44930:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 1102.217061] Lustre: 44930:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 1102.223098] Lustre: 44930:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 1102.240336] Lustre: 44930:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 1102.247540] Lustre: 44930:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 1105.410775] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 1111.491167] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1114.925346] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1120.586293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1120.642875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1120.867996] Lustre: lustre-OST0000: new disk, initializing [ 1120.876466] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1120.954632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1125.775299] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 1128.997414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1129.015640] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1129.170593] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1132.108446] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1135.182376] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1135.346701] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1138.389540] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1138.616323] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1144.803541] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1144.805088] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1144.806921] LustreError: Skipped 1 previous similar message [ 1144.810821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1144.819765] Lustre: Skipped 1 previous similar message [ 1146.313194] LustreError: 47076:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1146.496103] Lustre: server umount lustre-OST0000 complete [ 1149.931316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1149.936637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1149.945365] Lustre: Skipped 1 previous similar message [ 1149.955355] Lustre: Skipped 2 previous similar messages [ 1155.049344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1155.053097] Lustre: Skipped 1 previous similar message [ 1155.793056] LustreError: 47277:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1155.801467] LustreError: 47277:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1155.854379] Lustre: server umount lustre-MDT0000 complete [ 1158.473434] LustreError: 45882:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761238594 with bad export cookie 16441475024735457903 [ 1158.480420] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1158.481548] LustreError: 45882:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1158.593888] LustreError: 47479:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1158.595890] LustreError: 47479:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1158.779411] Lustre: server umount lustre-MDT0001 complete [ 1165.965719] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 12:56:40 (1761238600) [ 1170.949795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1171.297160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1173.968144] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 1176.546982] LustreError: 48199:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1176.572633] LustreError: 48199:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1181.066108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1181.305190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1184.275505] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 1188.484538] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1191.226985] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1196.303283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1196.621378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1200.731938] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 1205.045988] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1209.152716] LustreError: 50395:0:(llog.c:441:llog_init_handle()) llog type is not specified! [ 1209.884820] Lustre: Modifying parameter general.timeout=1 in log params [ 1211.435612] Lustre: Modifying parameter general.timeout=3 in log params [ 1211.437571] Lustre: Skipped 1 previous similar message [ 1213.875131] Lustre: Modifying parameter general.timeout=6 in log params [ 1213.878600] Lustre: Skipped 2 previous similar messages [ 1217.986565] Lustre: Modifying parameter general.timeout=12 in log params [ 1217.988856] Lustre: Skipped 5 previous similar messages [ 1226.112117] Lustre: Modifying parameter general.timeout=24 in log params [ 1226.115327] Lustre: Skipped 11 previous similar messages [ 1242.693439] Lustre: Modifying parameter general.timeout=50 in log params [ 1242.697765] Lustre: Skipped 25 previous similar messages [ 1274.752467] Lustre: Modifying parameter general.timeout=105 in log params [ 1274.756403] Lustre: Skipped 54 previous similar messages [ 1299.202230] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 12:58:54 (1761238734) [ 1301.503928] Lustre: 57314:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1302.876453] Lustre: 57412:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1305.439667] Lustre: 57625:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1305.446936] Lustre: 57625:0:(mgs_llog.c:1346:mgs_modify_param()) Skipped 1 previous similar message [ 1325.672061] Lustre: 59177:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1325.679225] Lustre: 59177:0:(mgs_llog.c:1346:mgs_modify_param()) Skipped 2 previous similar messages [ 1325.700702] LustreError: 59177:0:(mgs_handler.c:1126:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1328.545641] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 12:59:23 (1761238763) [ 1337.955579] LustreError: 59874:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1337.968967] LustreError: 59874:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1338.055135] Lustre: server umount lustre-MDT0000 complete [ 1339.873162] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1339.874926] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1339.878644] LustreError: 48912:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1339.886271] Lustre: Skipped 1 previous similar message [ 1339.899888] LustreError: 48912:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1340.532815] LustreError: 48179:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761238776 with bad export cookie 16441475024735458827 [ 1340.533585] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1340.541686] LustreError: 48179:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1340.859586] Lustre: server umount lustre-MDT0001 complete [ 1354.253988] LustreError: 60275:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1354.256048] LustreError: 60275:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1354.421455] Lustre: server umount lustre-OST0000 complete [ 1360.719670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1363.609421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1365.920537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1368.203703] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1372.428492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1372.600601] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1372.623590] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1372.629077] Lustre: Skipped 47 previous similar messages [ 1372.792036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1374.837477] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 1380.895511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1381.065587] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1381.091732] Lustre: 61726:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1385.983426] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 1390.722367] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1393.686954] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1398.708337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1398.840184] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1403.781590] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 1408.685715] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1410.101104] Lustre: 63958:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1414.627225] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1414.645384] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1414.667934] Lustre: Skipped 1 previous similar message [ 1414.674807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1416.241645] LustreError: 64205:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1416.246073] LustreError: 64205:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1416.383591] Lustre: server umount lustre-OST0000 complete [ 1418.889686] Lustre: server umount lustre-MDT0000 complete [ 1419.746794] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1419.759340] LustreError: 61732:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1419.771330] LustreError: 61732:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1421.391832] LustreError: 63960:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761238857 with bad export cookie 16441475024735461452 [ 1421.397444] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1421.397743] LustreError: 63960:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1431.209986] Lustre: server umount lustre-MDT0001 complete [ 1438.376954] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 1440.181649] Key type lgssc unregistered [ 1440.368927] LNet: 65161:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1440.375144] LNetError: 65161:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1441.448991] LNet: Removed LNI 192.168.206.120@tcp [ 1441.976910] Key type .llcrypt unregistered [ 1441.979615] Key type ._llcrypt unregistered [ 1457.748357] Key type ._llcrypt registered [ 1457.753375] Key type .llcrypt registered [ 1457.834646] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 1458.578151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1458.626187] alg: No test for adler32 (adler32-zlib) [ 1459.704259] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 1459.883180] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 1461.519196] Key type lgssc registered [ 1462.672852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1469.510745] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1469.541190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1470.979569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1473.833916] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 1479.032313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1479.367651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1481.857985] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 1486.182707] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1489.162886] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1494.630704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1495.096765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1498.934510] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 1503.451974] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1509.186329] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1514.466224] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1514.471294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1514.476877] Lustre: Skipped 2 previous similar messages [ 1514.479996] Lustre: Skipped 1 previous similar message [ 1519.585733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1519.597354] Lustre: Skipped 3 previous similar messages [ 1520.161393] LustreError: 69244:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1520.234066] Lustre: server umount lustre-MDT0000 complete [ 1523.195849] LustreError: 68888:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761238959 with bad export cookie 17223197945700741146 [ 1523.198516] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1523.205536] LustreError: 68888:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1523.306193] LustreError: 69447:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1523.308087] LustreError: 69447:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1523.441989] Lustre: server umount lustre-MDT0001 complete [ 1535.780666] LustreError: 69648:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1535.787834] LustreError: 69648:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1535.985952] Lustre: server umount lustre-OST0000 complete [ 1543.139975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1545.934500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1548.757539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1551.364561] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1556.317751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1556.523683] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1556.543924] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1556.733956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1559.547865] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 1565.553576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1565.606267] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1565.625879] Lustre: 71097:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1565.990941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1568.757778] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 1572.736420] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1575.140335] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1579.881434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1580.000238] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1580.303730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1584.557473] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 1588.013237] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1589.126696] Lustre: 73340:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1590.408332] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1596.386058] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1596.394974] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1596.400103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1596.422466] Lustre: Skipped 1 previous similar message [ 1596.440157] Lustre: Skipped 1 previous similar message [ 1599.159110] LustreError: 73590:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1599.163312] LustreError: 73590:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1599.347778] Lustre: server umount lustre-OST0000 complete [ 1606.624397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1606.625167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1606.645691] Lustre: Skipped 1 previous similar message [ 1608.498242] LustreError: 73798:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1608.513057] LustreError: 73798:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1608.620475] Lustre: server umount lustre-MDT0000 complete [ 1611.693351] LustreError: 73343:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761239048 with bad export cookie 17223197945700741979 [ 1611.701210] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1612.140413] Lustre: server umount lustre-MDT0001 complete [ 1620.986905] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 1623.337845] Key type lgssc unregistered [ 1623.608324] LNet: 74530:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1623.618270] LNetError: 74530:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1623.644337] LNet: Removed LNI 192.168.206.120@tcp [ 1624.351545] Key type .llcrypt unregistered [ 1624.353721] Key type ._llcrypt unregistered [ 1645.897948] Key type ._llcrypt registered [ 1645.900915] Key type .llcrypt registered [ 1645.973526] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 1646.793199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1646.858503] alg: No test for adler32 (adler32-zlib) [ 1647.967049] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 1648.174953] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 1649.831154] Key type lgssc registered [ 1650.704690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1657.031626] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1657.054099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1658.468858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1661.666987] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 1667.768744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1668.173722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1670.902858] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 1674.917507] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1677.667966] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1682.196155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1682.630192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1686.649260] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 1690.183772] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1698.190235] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1698.925989] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1700.413139] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1700.415795] Lustre: Skipped 1 previous similar message [ 1702.475060] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1702.479868] Lustre: Skipped 2 previous similar messages [ 1707.055785] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1707.057915] Lustre: Skipped 5 previous similar messages [ 1715.204673] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1715.207469] Lustre: Skipped 11 previous similar messages [ 1731.700698] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1731.704514] Lustre: Skipped 23 previous similar messages [ 1763.705229] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1763.716846] Lustre: Skipped 38 previous similar messages [ 1779.898803] Lustre: 83693:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1782.092830] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 13:06:56 (1761239216) [ 1783.465338] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1785.164430] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 13:06:59 (1761239219) [ 1790.432894] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1790.439489] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1790.441378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1790.444923] Lustre: Skipped 1 previous similar message [ 1790.449684] Lustre: Skipped 1 previous similar message [ 1794.306136] LustreError: 83988:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1794.582418] Lustre: server umount lustre-OST0000 complete [ 1800.676075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1800.684791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1800.696668] Lustre: Skipped 1 previous similar message [ 1803.681038] LustreError: 84189:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1803.684661] LustreError: 84189:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1803.758231] Lustre: server umount lustre-MDT0000 complete [ 1805.795362] LustreError: 77551:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1805.813466] LustreError: 77551:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1806.672056] LustreError: 75904:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761239242 with bad export cookie 7004004297934679314 [ 1806.673702] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1806.708250] LustreError: 75904:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1806.827496] LustreError: 84391:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1806.830203] LustreError: 84391:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1807.094544] Lustre: server umount lustre-MDT0001 complete [ 1816.047139] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 1818.281884] Key type lgssc unregistered [ 1818.530940] LNet: 84921:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1818.541707] LNetError: 84921:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1819.557780] LNet: Removed LNI 192.168.206.120@tcp [ 1820.350828] Key type .llcrypt unregistered [ 1820.352979] Key type ._llcrypt unregistered [ 1823.730394] Key type ._llcrypt registered [ 1823.732167] Key type .llcrypt registered [ 1823.854347] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1827.590877] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules [ 1828.265797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1828.296075] alg: No test for adler32 (adler32-zlib) [ 1829.368461] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 1829.549842] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 1831.216678] Key type lgssc registered [ 1832.176215] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1833.158075] LustreError: 86085:0:(lu_object.c:1438:lu_context_key_register()) cfs_race id 60d sleeping [ 1848.676356] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 1855.594198] LustreError: 86085:0:(lu_object.c:1438:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1858.435904] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1858.447827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1859.972690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1862.872404] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 1868.686804] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 13:08:23 (1761239303) [ 1870.308291] LustreError: 87092:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1870.315465] LustreError: 87092:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1871.437443] LustreError: 87124:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1871.442570] LustreError: 87124:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1871.446817] LustreError: 87124:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 11, retries 0, failed: rc = -5 [ 1871.528824] LustreError: 87900:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1871.672809] Lustre: server umount lustre-MDT0000 complete [ 1881.084852] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 1883.420712] Key type lgssc unregistered [ 1883.668833] LNet: 88586:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1883.680323] LNetError: 88586:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1884.713122] LNet: Removed LNI 192.168.206.120@tcp [ 1885.539838] Key type .llcrypt unregistered [ 1885.542261] Key type ._llcrypt unregistered [ 1898.852752] Key type ._llcrypt registered [ 1898.854317] Key type .llcrypt registered [ 1898.992120] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 1899.709814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1899.723985] alg: No test for adler32 (adler32-zlib) [ 1900.672836] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 1900.802882] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 1902.431234] Key type lgssc registered [ 1903.206960] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1909.001647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1909.028355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1910.408311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1913.371667] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 1919.972415] LustreError: 89764:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1919.980337] LustreError: 89764:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1919.994854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1920.591670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1923.412710] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 1927.604811] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1930.308857] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1936.160927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1936.936292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1942.084493] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 1946.824370] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1992.923348] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 13:10:27 (1761239427) [ 1994.522225] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1996.052489] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 13:10:30 (1761239430) [ 2001.381258] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2001.388312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2001.400574] Lustre: Skipped 2 previous similar messages [ 2001.405726] Lustre: Skipped 1 previous similar message [ 2005.232085] LustreError: 92692:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2005.320534] Lustre: server umount lustre-MDT0000 complete [ 2006.497939] LustreError: 90479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2006.516678] LustreError: 90479:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2008.350866] LustreError: 89745:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761239444 with bad export cookie 641790354637859092 [ 2008.363800] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2008.499205] LustreError: 92894:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2008.501173] LustreError: 92894:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2008.707208] Lustre: server umount lustre-MDT0001 complete [ 2022.024361] LustreError: 93095:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2022.031130] LustreError: 93095:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2022.315249] Lustre: server umount lustre-OST0000 complete [ 2032.932858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2033.396672] LustreError: 93724:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2033.612865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2037.784557] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 2038.754318] LustreError: 93724:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2042.852992] LustreError: 93724:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2045.748944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2046.470335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2049.581631] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 2054.987843] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2058.029982] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2063.814453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2069.931611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2070.007689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2070.106480] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2070.115286] LustreError: 93714:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2070.154681] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2070.160963] LustreError: 95588:0:(tgt_mount.c:2482:server_fill_super()) Unable to start targets: -98 [ 2070.166780] LustreError: 95588:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 2070.170712] LustreError: 95588:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 2070.352410] Lustre: server umount lustre-OST0000 complete [ 2070.354653] LustreError: 95588:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2077.224551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2077.425261] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2077.433337] LustreError: 93714:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2077.448204] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2077.453340] LustreError: 96049:0:(tgt_mount.c:2482:server_fill_super()) Unable to start targets: -98 [ 2077.458659] LustreError: 96049:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 2077.462509] LustreError: 96049:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 2077.569784] Lustre: server umount lustre-OST0000 complete [ 2077.574447] LustreError: 96049:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2079.846344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2084.645329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2084.765557] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2084.769312] Lustre: Found index 0 for lustre-OST0000, updating log [ 2084.774474] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2084.822208] Lustre: lustre-OST0000: new disk, initializing [ 2084.828730] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2085.023826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2089.233554] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 2092.241586] Lustre: Failing over lustre-OST0000 [ 2092.281100] LustreError: 97138:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2092.286634] LustreError: 97138:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2092.419548] Lustre: server umount lustre-OST0000 complete [ 2097.632543] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2097.641463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2097.654270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2097.657550] Lustre: Skipped 1 previous similar message [ 2098.655924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2098.661192] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2101.476252] LustreError: 97339:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2101.478424] LustreError: 97339:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2101.562425] Lustre: server umount lustre-MDT0000 complete [ 2102.755194] LustreError: 93720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2102.780532] LustreError: 93720:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2104.259907] LustreError: 96590:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761239540 with bad export cookie 641790354637861836 [ 2104.261136] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2104.266310] LustreError: 96590:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2104.551265] Lustre: server umount lustre-MDT0001 complete [ 2110.906832] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 13:12:25 (1761239545) [ 2121.635503] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 2128.288100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2128.523499] LustreError: 98909:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2128.609776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2131.089283] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 2136.243310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2138.861971] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 2140.715894] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2145.527779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2149.648979] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 2155.532967] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2155.581188] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2155.617538] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2155.685933] Lustre: lustre-OST0001: new disk, initializing [ 2155.688845] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2159.930272] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 2163.289453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2163.302426] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2163.402054] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2166.128130] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2172.425084] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2182.252380] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.20@tcp (stopping) [ 2182.257648] Lustre: Skipped 1 previous similar message [ 2184.166823] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2184.189941] Lustre: Skipped 1 previous similar message [ 2186.868582] LustreError: 102150:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2186.872512] LustreError: 102150:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2186.947751] Lustre: server umount lustre-MDT0001 complete [ 2187.340504] LustreError: 98904:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2187.354106] LustreError: 98904:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2188.707348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2193.489751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2193.547747] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2193.550200] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2193.588714] Lustre: 98898:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2194.122449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2194.126112] Lustre: Skipped 3 previous similar messages [ 2195.512371] LustreError: 102779:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2195.525719] LustreError: 102779:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2195.532234] LustreError: 98900:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2195.624107] LustreError: 102789:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.206.120@tcp: cfg command failed: rc = -17 [ 2195.629359] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2195.629359] [ 2197.072972] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 2199.533366] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2204.782957] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 13:13:59 (1761239639) [ 2218.873964] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 2232.034899] Lustre: 104761:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2251.018268] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2254.051917] Lustre: 105894:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2259.278787] Lustre: 98905:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 505 < left 1007, rollback = 2 [ 2259.283565] Lustre: 98905:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 4/16/7, destroy: 0/0/0 [ 2259.293266] Lustre: 98905:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 8/1007/0 [ 2259.296278] Lustre: 98905:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/100/0, punch: 0/0/0, quota 1/3/0 [ 2259.304385] Lustre: 98905:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 12/236/4, delete: 0/0/0 [ 2259.308576] Lustre: 98905:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 4/4/0, ref_del: 0/0/0 [ 2290.532259] Lustre: 98903:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 993, rollback = 2 [ 2290.537188] Lustre: 98903:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 8 previous similar messages [ 2290.541838] Lustre: 98903:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/5, destroy: 0/0/0 [ 2290.546599] Lustre: 98903:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 2290.554478] Lustre: 98903:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 7/993/0 [ 2290.569511] Lustre: 98903:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 2290.575232] Lustre: 98903:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 5/58/0, punch: 0/0/0, quota 1/3/0 [ 2290.579745] Lustre: 98903:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 2290.583969] Lustre: 98903:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 10/203/4, delete: 0/0/0 [ 2290.588206] Lustre: 98903:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 2290.593155] Lustre: 98903:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 4/4/0, ref_del: 0/0/0 [ 2290.597371] Lustre: 98903:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 2337.249185] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2337.252771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2337.263946] Lustre: Skipped 1 previous similar message [ 2337.272538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2337.275114] Lustre: Skipped 3 previous similar messages [ 2339.254536] LustreError: 106356:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2339.276590] LustreError: 106356:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2339.328608] Lustre: server umount lustre-MDT0000 complete [ 2341.854159] LustreError: 101886:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761239778 with bad export cookie 641790354637862760 [ 2341.856171] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2341.865991] LustreError: 101886:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2342.087618] Lustre: server umount lustre-MDT0001 complete [ 2355.040097] Lustre: server umount lustre-OST0000 complete [ 2368.424617] Lustre: server umount lustre-OST0001 complete [ 2373.183335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2379.022964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2379.782566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2385.882192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2392.012031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2392.687740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2409.327749] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 2418.282554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2418.297309] Lustre: lustre-MDT0000: reset Object Index mappings [ 2418.695606] LustreError: 109840:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2418.721191] LustreError: 109840:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2418.804991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2422.087071] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 2429.258704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2429.284203] Lustre: lustre-MDT0001: reset Object Index mappings [ 2429.412855] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2429.416719] Lustre: Skipped 3 previous similar messages [ 2432.668412] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 2434.768784] Lustre: 110955:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2440.409694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2444.208827] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 2446.042681] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 2453.252790] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2454.775358] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 2454.777084] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 2454.868236] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 2458.489833] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 2463.037741] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2465.014340] Lustre: 112810:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2506.072713] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 13:19:00 (1761239940) [ 2536.929093] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2536.929766] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2536.941391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2536.952219] Lustre: Skipped 6 previous similar messages [ 2536.963053] Lustre: Skipped 3 previous similar messages [ 2539.251514] LustreError: 113164:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2539.265277] LustreError: 113164:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 2539.354027] Lustre: server umount lustre-MDT0000 complete [ 2542.086878] LustreError: 109819:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761239978 with bad export cookie 641790354637909632 [ 2542.093691] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2542.102132] LustreError: 109819:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2547.168731] LustreError: 109835:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2547.177332] LustreError: 109835:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 2548.523716] Lustre: server umount lustre-MDT0001 complete [ 2556.706950] Lustre: server umount lustre-OST0000 complete [ 2573.279136] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2573.551338] Lustre: server umount lustre-OST0001 complete [ 2577.618153] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [ 2582.805291] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 2589.761523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2595.307967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2600.371665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2605.813640] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2611.368638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2611.430867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2611.601115] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2611.623429] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2611.684422] Lustre: lustre-MDT0000: new disk, initializing [ 2611.787224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2611.791781] Lustre: Skipped 3 previous similar messages [ 2611.809146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2615.018517] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 2620.575490] LustreError: 116523:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2620.581036] LustreError: 116523:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 2620.734888] Lustre: server umount lustre-MDT0000 complete [ 2622.441775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2626.095363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2626.124131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2626.281257] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2626.297880] Lustre: 117052:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2626.307962] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2628.613813] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 2631.661833] Lustre: server umount lustre-MDT0000 complete [ 2637.602253] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 13:21:12 (1761240072) [ 2638.756872] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2640.127216] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 13:21:15 (1761240075) [ 2652.742356] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 2660.813960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2663.784781] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 2669.732787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2669.776299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2669.850597] Lustre: 119070:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2669.898168] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2669.903854] Lustre: Skipped 1 previous similar message [ 2669.977096] Lustre: lustre-MDT0001: new disk, initializing [ 2670.082077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2670.088175] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2670.134829] Lustre: 119811:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 2670.142453] Lustre: 119811:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 8 previous similar messages [ 2670.149854] Lustre: 119811:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 2670.156484] Lustre: 119811:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 2670.161338] Lustre: 119811:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 2670.166358] Lustre: 119811:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 2670.171562] Lustre: 119811:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 2670.178613] Lustre: 119811:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 2670.183324] Lustre: 119811:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 2670.187453] Lustre: 119811:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 2670.198526] Lustre: 119811:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 2670.207918] Lustre: 119811:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 2672.420877] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 2675.989803] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2680.459779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2680.519447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2680.680044] Lustre: lustre-OST0000: new disk, initializing [ 2680.682976] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2682.309087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2682.322741] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2682.455731] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2685.200488] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 2694.407113] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2694.467013] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2694.592754] Lustre: lustre-OST0001: new disk, initializing [ 2694.601094] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2696.069603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2696.078682] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2696.174956] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2698.872138] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 2707.071909] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2710.378686] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2715.619450] Lustre: 119077:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 504 < left 1021, rollback = 2 [ 2715.624109] Lustre: 119077:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 1 previous similar message [ 2715.627251] Lustre: 119077:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 4/16/8, destroy: 0/0/0 [ 2715.630472] Lustre: 119077:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 1 previous similar message [ 2715.634283] Lustre: 119077:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 4/4/0, xattr_set: 9/1021/0 [ 2715.637991] Lustre: 119077:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 2715.644516] Lustre: 119077:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/153/0, punch: 0/0/0, quota 1/3/0 [ 2715.648523] Lustre: 119077:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 2715.651189] Lustre: 119077:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 13/232/4, delete: 0/0/0 [ 2715.655623] Lustre: 119077:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 2715.657858] Lustre: 119077:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 5/5/0, ref_del: 0/0/0 [ 2715.659869] Lustre: 119077:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3043.506872] Lustre: 127862:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3043.515917] Lustre: 127862:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 8 previous similar messages [ 3043.526925] Lustre: 127862:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3043.539549] Lustre: 127862:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 3043.547830] Lustre: 127862:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3043.551607] Lustre: 127862:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 3043.556098] Lustre: 127862:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3043.560680] Lustre: 127862:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 3043.565373] Lustre: 127862:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3043.569967] Lustre: 127862:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 3043.573221] Lustre: 127862:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3043.577507] Lustre: 127862:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 3273.457715] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 13:31:47 (1761240707) [ 3288.033662] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3288.037780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3288.048424] Lustre: Skipped 5 previous similar messages [ 3288.052053] Lustre: Skipped 4 previous similar messages [ 3293.157219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3293.160846] Lustre: Skipped 4 previous similar messages [ 3293.723341] LustreError: 128088:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3293.731462] LustreError: 128088:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3293.932565] Lustre: server umount lustre-MDT0000 complete [ 3297.105589] LustreError: 119063:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761240733 with bad export cookie 641790354637973353 [ 3297.107440] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3297.123495] LustreError: 119063:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3298.276482] LustreError: 122235:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3298.281847] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3298.304475] LustreError: 122235:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3298.314725] Lustre: Skipped 1 previous similar message [ 3302.308916] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3302.311167] Lustre: Skipped 2 previous similar messages [ 3303.763301] Lustre: server umount lustre-MDT0001 complete [ 3313.419287] Lustre: server umount lustre-OST0000 complete [ 3323.494661] Lustre: server umount lustre-OST0001 complete [ 3329.017297] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [ 3337.643205] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 3346.872223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3353.006579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3359.217097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3364.378688] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3369.515063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3369.558921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3369.705800] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3369.726439] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3369.771617] Lustre: lustre-MDT0000: new disk, initializing [ 3369.822766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3369.828192] Lustre: Skipped 5 previous similar messages [ 3369.849342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3372.793889] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 3382.739590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3382.838122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3382.929198] Lustre: 130877:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3383.025436] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3383.029883] Lustre: Skipped 1 previous similar message [ 3383.145786] Lustre: lustre-MDT0001: new disk, initializing [ 3383.293324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3383.301857] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3387.259678] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 3394.973234] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3398.528849] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3404.513592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3404.579615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3404.809784] Lustre: lustre-OST0000: new disk, initializing [ 3404.813452] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3406.736293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3406.755345] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3406.834266] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3410.094622] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 3417.461297] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3426.714650] Lustre: lustre-MDD0000: changelog on [ 3429.065670] Lustre: lustre-MDD0001: changelog on [ 3432.428817] Lustre: 130886:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 513 < left 539, rollback = 2 [ 3432.438324] Lustre: 130886:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 255 previous similar messages [ 3432.443749] Lustre: 130886:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/1, destroy: 0/0/0 [ 3432.449353] Lustre: 130886:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 255 previous similar messages [ 3432.453397] Lustre: 130886:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 4/4/0, xattr_set: 9/539/0 [ 3432.460671] Lustre: 130886:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 255 previous similar messages [ 3432.473271] Lustre: 130886:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 10/93/1, punch: 0/0/0, quota 1/3/0 [ 3432.480272] Lustre: 130886:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 255 previous similar messages [ 3432.496339] Lustre: 130886:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 4/67/1, delete: 0/0/0 [ 3432.499766] Lustre: 130886:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 255 previous similar messages [ 3432.506188] Lustre: 130886:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3432.515352] Lustre: 130886:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 255 previous similar messages [ 3448.428349] Lustre: 130885:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 515 < left 525, rollback = 2 [ 3448.436700] Lustre: 130885:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 4160 previous similar messages [ 3448.456532] Lustre: 130885:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 2/8/0, destroy: 0/0/0 [ 3448.467788] Lustre: 130885:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 4163 previous similar messages [ 3448.481877] Lustre: 130885:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 7/525/0 [ 3448.489871] Lustre: 130885:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 4163 previous similar messages [ 3448.504429] Lustre: 130885:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/39/0, punch: 0/0/0, quota 1/3/0 [ 3448.512165] Lustre: 130885:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 4163 previous similar messages [ 3448.517546] Lustre: 130885:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 3/50/0, delete: 0/0/0 [ 3448.521206] Lustre: 130885:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 4163 previous similar messages [ 3448.525145] Lustre: 130885:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3448.529194] Lustre: 130885:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 4163 previous similar messages [ 3480.433696] Lustre: 130885:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 525, rollback = 2 [ 3480.438314] Lustre: 130885:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 7748 previous similar messages [ 3480.457219] Lustre: 130886:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 2/8/0, destroy: 0/0/0 [ 3480.460638] Lustre: 130886:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 7751 previous similar messages [ 3480.489507] Lustre: 130886:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 7/525/0 [ 3480.499686] Lustre: 130886:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 7757 previous similar messages [ 3480.504326] Lustre: 130886:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/39/0, punch: 0/0/0, quota 1/3/0 [ 3480.514527] Lustre: 130886:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 7757 previous similar messages [ 3480.523397] Lustre: 130886:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 3/50/0, delete: 0/0/0 [ 3480.532813] Lustre: 130886:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 7757 previous similar messages [ 3480.545176] Lustre: 130886:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3480.562979] Lustre: 130886:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 7757 previous similar messages [ 3544.437499] Lustre: 130884:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 525, rollback = 2 [ 3544.442018] Lustre: 130884:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 5003 previous similar messages [ 3544.461412] Lustre: 133963:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 2/8/0, destroy: 0/0/0 [ 3544.464188] Lustre: 133963:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 5003 previous similar messages [ 3544.489954] Lustre: 133963:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 7/525/0 [ 3544.498584] Lustre: 133963:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 5006 previous similar messages [ 3544.506691] Lustre: 133963:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/39/0, punch: 0/0/0, quota 1/3/0 [ 3544.512396] Lustre: 133963:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 5006 previous similar messages [ 3544.531763] Lustre: 133963:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 3/50/0, delete: 0/0/0 [ 3544.536785] Lustre: 133963:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 5009 previous similar messages [ 3544.552174] Lustre: 133963:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3544.555668] Lustre: 133963:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 5012 previous similar messages [ 3672.443445] Lustre: 130886:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 525, rollback = 2 [ 3672.446800] Lustre: 130886:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 22448 previous similar messages [ 3672.463844] Lustre: 130886:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 2/8/0, destroy: 0/0/0 [ 3672.467160] Lustre: 130886:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 22445 previous similar messages [ 3672.493236] Lustre: 130886:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 7/525/0 [ 3672.501132] Lustre: 130886:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 22442 previous similar messages [ 3672.511858] Lustre: 130886:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/39/0, punch: 0/0/0, quota 1/3/0 [ 3672.521124] Lustre: 130886:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 22442 previous similar messages [ 3672.532191] Lustre: 130886:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 3/50/0, delete: 0/0/0 [ 3672.537261] Lustre: 130886:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 22441 previous similar messages [ 3672.559677] Lustre: 130886:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3672.562851] Lustre: 130886:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 22439 previous similar messages [ 3928.445365] Lustre: 130884:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 525, rollback = 2 [ 3928.455327] Lustre: 130884:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 39914 previous similar messages [ 3928.471637] Lustre: 130885:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 2/8/1, destroy: 0/0/0 [ 3928.475223] Lustre: 130885:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 39914 previous similar messages [ 3928.501817] Lustre: 130884:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 7/525/0 [ 3928.505463] Lustre: 130884:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 39914 previous similar messages [ 3928.519860] Lustre: 130884:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/39/0, punch: 0/0/0, quota 1/3/0 [ 3928.526219] Lustre: 130884:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 39917 previous similar messages [ 3928.534427] Lustre: 130884:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 3/50/0, delete: 0/0/0 [ 3928.542903] Lustre: 130884:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 39915 previous similar messages [ 3928.575476] Lustre: 130884:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3928.592462] Lustre: 130884:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 39917 previous similar messages [ 4440.446422] Lustre: 130886:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 515 < left 525, rollback = 2 [ 4440.454596] Lustre: 130886:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 79406 previous similar messages [ 4440.475559] Lustre: 130885:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 2/8/0, destroy: 0/0/0 [ 4440.482755] Lustre: 130885:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 79406 previous similar messages [ 4440.512994] Lustre: 130886:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 7/525/0 [ 4440.517891] Lustre: 130886:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 79406 previous similar messages [ 4440.526065] Lustre: 130886:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/39/0, punch: 0/0/0, quota 1/3/0 [ 4440.532276] Lustre: 130886:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 79403 previous similar messages [ 4440.536972] Lustre: 130886:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 3/50/0, delete: 0/0/0 [ 4440.539913] Lustre: 130886:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 79403 previous similar messages [ 4440.578451] Lustre: 130885:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4440.583987] Lustre: 130885:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 79412 previous similar messages [ 5066.354192] Lustre: lustre-MDD0001: changelog off [ 5068.025378] Lustre: 133061:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5068.769952] Lustre: lustre-MDD0000: changelog off [ 5075.941193] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5075.941919] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5075.950170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5075.956601] Lustre: Skipped 1 previous similar message [ 5079.683635] LustreError: 140662:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5079.686671] LustreError: 140662:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 5079.932528] Lustre: server umount lustre-OST0000 complete [ 5083.113906] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5083.117108] LustreError: Skipped 1 previous similar message [ 5083.118992] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5083.124849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5083.126686] Lustre: Skipped 1 previous similar message [ 5086.179289] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5089.472328] Lustre: server umount lustre-MDT0000 complete [ 5091.302963] LustreError: 130890:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5091.313074] LustreError: 130890:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5093.194947] LustreError: 140865:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761242529 with bad export cookie 641790354638738285 [ 5093.197655] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5093.203241] LustreError: 140865:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5093.467503] Lustre: server umount lustre-MDT0001 complete [ 5100.066787] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 5102.200841] Key type lgssc unregistered [ 5102.485418] LNet: 141597:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5102.494919] LNetError: 141597:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5102.509668] LNet: Removed LNI 192.168.206.120@tcp [ 5103.204475] Key type .llcrypt unregistered [ 5103.208522] Key type ._llcrypt unregistered [ 5118.886976] Key type ._llcrypt registered [ 5118.888677] Key type .llcrypt registered [ 5118.974722] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [ 5130.514895] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 5131.447711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5131.466741] alg: No test for adler32 (adler32-zlib) [ 5132.451140] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 5132.666024] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 5134.319188] Key type lgssc registered [ 5135.259555] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5143.198253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5149.200035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5154.247764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5159.852763] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5166.848835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5166.975442] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5167.029398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5168.401870] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5168.469476] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5168.546306] Lustre: lustre-MDT0000: new disk, initializing [ 5168.704647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5168.731244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5173.479845] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 5185.383566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5185.518970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5185.638261] Lustre: 144756:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5185.683132] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5185.685726] Lustre: Skipped 1 previous similar message [ 5185.764398] Lustre: lustre-MDT0001: new disk, initializing [ 5185.873339] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5185.947320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5185.971258] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5186.111574] Lustre: 145552:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 5186.122943] Lustre: 145552:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 5186.136056] Lustre: 145552:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 5186.142078] Lustre: 145552:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 5186.151078] Lustre: 145552:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 5186.160421] Lustre: 145552:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 5190.117280] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 5196.738685] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5199.641531] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5205.145491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5205.212669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5205.443390] Lustre: lustre-OST0000: new disk, initializing [ 5205.445808] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5205.529737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5209.419857] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 5212.181475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5212.187404] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5212.256904] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5214.499422] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5218.509451] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_module obdecho/obdecho [ 5219.771909] LustreError: 147377:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5219.778076] LustreError: 147377:0:(obd_config.c:783:class_setup()) setup ec failed (-22) [ 5221.017438] LustreError: 147473:0:(echo_client.c:1880:echo_md_handler()) Can not resolve the path : rc = -2 [ 5232.097099] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5232.108810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5232.117164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5232.615341] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5232.619701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5232.642868] Lustre: Skipped 1 previous similar message [ 5232.664795] Lustre: Skipped 1 previous similar message [ 5235.501449] LustreError: 147813:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5235.614920] Lustre: server umount lustre-MDT0000 complete [ 5238.937451] LustreError: 145536:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761242675 with bad export cookie 11676478698500284708 [ 5238.938889] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5238.944023] LustreError: 145536:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5238.952750] LustreError: 148014:0:(client.c:1370:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000002b279cfe x1846796687022464/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5238.952870] LustreError: 148014:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x20000000a:0x0:0x0]: rc = -5 [ 5238.953038] LustreError: 148014:0:(llog_cat.c:443:llog_cat_close()) lustre-MDT0000-osp-MDT0001: failure destroying log during cleanup: rc = -5 [ 5239.071171] LustreError: 148014:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5239.075771] LustreError: 148014:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5239.253256] Lustre: server umount lustre-MDT0001 complete [ 5252.037036] LustreError: 148215:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5252.038888] LustreError: 148215:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5252.168800] Lustre: server umount lustre-OST0000 complete [ 5267.316628] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [ 5274.871973] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 5281.631404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5287.224103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5292.088852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5296.728423] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5301.464421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5301.532786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5301.731749] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5301.761435] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5301.820961] Lustre: lustre-MDT0000: new disk, initializing [ 5301.896901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5301.913285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5305.243899] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 5314.692945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5314.766237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5314.858543] Lustre: 151160:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5314.929532] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5314.933661] Lustre: Skipped 1 previous similar message [ 5314.977349] Lustre: lustre-MDT0001: new disk, initializing [ 5315.109470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5315.191427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5315.211228] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5315.316227] Lustre: 151957:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 5315.343151] Lustre: 151957:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 1 previous similar message [ 5315.351133] Lustre: 151957:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 5315.364362] Lustre: 151957:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5315.375708] Lustre: 151957:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 5315.396242] Lustre: 151957:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5315.409338] Lustre: 151957:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 5315.425822] Lustre: 151957:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5315.459627] Lustre: 151957:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 5315.461956] Lustre: 151957:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5315.464374] Lustre: 151957:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 5315.490343] Lustre: 151957:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5318.798514] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 5324.745477] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5328.011676] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5333.439240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5333.522260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5333.805603] Lustre: lustre-OST0000: new disk, initializing [ 5333.809880] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5333.891646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5338.401817] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 5342.796713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5342.818912] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5342.962372] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5345.996860] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5349.601325] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5349.861056] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5352.920156] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5353.111842] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5358.561053] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5358.566185] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5358.586940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5360.719145] LustreError: 154105:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5360.728241] LustreError: 154105:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5360.900717] Lustre: server umount lustre-OST0000 complete [ 5366.241918] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5366.254993] LustreError: Skipped 1 previous similar message [ 5366.257079] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5366.280032] Lustre: Skipped 1 previous similar message [ 5366.285991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5366.293348] Lustre: Skipped 1 previous similar message [ 5370.826315] LustreError: 154306:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5370.834833] LustreError: 154306:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5370.973315] Lustre: server umount lustre-MDT0000 complete [ 5373.928707] LustreError: 153278:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5373.945333] LustreError: 153278:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5374.741554] LustreError: 151153:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761242811 with bad export cookie 11676478698500285709 [ 5374.750904] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5374.763016] LustreError: 151153:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5375.320185] Lustre: server umount lustre-MDT0001 complete [ 5378.609808] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 14:06:53 (1761242813) [ 5380.175094] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 5381.743222] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 14:06:56 (1761242816) [ 5388.509715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5388.895673] LustreError: 155180:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5388.994972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5392.051111] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 5394.402468] LustreError: 155180:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5398.474564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5401.684816] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 5405.725846] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5408.170593] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5412.976463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5417.013366] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 5420.398814] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5421.085955] Lustre: 156294:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 504 < left 1021, rollback = 2 [ 5421.096394] Lustre: 156294:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 1 previous similar message [ 5421.101139] Lustre: 156294:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 4/16/8, destroy: 0/0/0 [ 5421.104625] Lustre: 156294:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5421.112034] Lustre: 156294:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 4/4/0, xattr_set: 9/1021/0 [ 5421.117900] Lustre: 156294:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5421.122743] Lustre: 156294:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/153/0, punch: 0/0/0, quota 1/3/0 [ 5421.127228] Lustre: 156294:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5421.131606] Lustre: 156294:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 13/232/4, delete: 0/0/0 [ 5421.136474] Lustre: 156294:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5421.143475] Lustre: 156294:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 5/5/0, ref_del: 0/0/0 [ 5421.147878] Lustre: 156294:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5431.383979] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.20@tcp (stopping) [ 5431.387120] Lustre: Skipped 2 previous similar messages [ 5431.893376] LustreError: 157428:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5431.895531] LustreError: 157428:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5431.935538] Lustre: server umount lustre-MDT0000 complete [ 5434.296980] LustreError: 155160:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761242870 with bad export cookie 11676478698500286626 [ 5434.305029] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5434.307555] LustreError: 155160:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5434.854610] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5434.858055] LustreError: 155175:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5434.865709] Lustre: Skipped 2 previous similar messages [ 5439.969699] LustreError: 156294:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5439.970055] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5439.979315] LustreError: 156294:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5439.981532] Lustre: Skipped 2 previous similar messages [ 5440.633287] Lustre: server umount lustre-MDT0001 complete [ 5445.559990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5455.913554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5456.161397] LustreError: 158703:0:(llog.c:1616:llog_backup()) MGC192.168.206.120@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5456.166561] Lustre: 158703:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.206.120@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 5456.971153] LustreError: 158711:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5474.783959] LustreError: 158709:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5474.803172] LustreError: 158709:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 5508.166634] LustreError: 158727:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5508.189297] LustreError: 158727:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 24 previous similar messages [ 5517.280987] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 5517.282930] Lustre: 158703:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.206.120@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 5544.927373] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 5544.930346] Lustre: 155171:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.206.120@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 5557.812501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5557.817434] Lustre: Skipped 2 previous similar messages [ 5559.847354] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5561.018651] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 5565.407175] LustreError: 155171:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 5567.575902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5572.684369] LustreError: 158711:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5572.700105] LustreError: 158711:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 44 previous similar messages [ 5579.232959] LustreError: 142965:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000189bef17 x1846796687109248/t0(0) o250->MGC192.168.206.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5583.533897] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 5584.890286] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5588.217485] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5590.017884] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5593.416636] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5594.993665] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5610.465365] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5610.466654] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5610.470939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5610.482573] Lustre: Skipped 2 previous similar messages [ 5611.079754] LustreError: 160209:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5611.087521] LustreError: 160209:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5611.178399] Lustre: server umount lustre-MDT0000 complete [ 5613.966386] LustreError: 155159:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761243050 with bad export cookie 11676478698500293143 [ 5613.970654] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5613.978575] LustreError: 155159:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5614.222377] Lustre: server umount lustre-MDT0001 complete [ 5627.281713] Lustre: server umount lustre-OST0000 complete [ 5643.175289] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [ 5650.565753] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 5658.336705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5664.578030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5671.708021] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5678.604225] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5684.956931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5685.024053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5685.204224] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5685.238859] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5685.334071] Lustre: lustre-MDT0000: new disk, initializing [ 5685.452488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5685.454968] Lustre: Skipped 1 previous similar message [ 5685.471388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5689.150034] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 5697.785778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5697.861464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5697.943053] Lustre: 163561:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5698.178989] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5698.239665] Lustre: 164360:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 5698.246892] Lustre: 164360:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 3 previous similar messages [ 5698.250507] Lustre: 164360:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 5698.253426] Lustre: 164360:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 5698.259726] Lustre: 164360:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 5698.267062] Lustre: 164360:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 5698.273680] Lustre: 164360:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 5698.279947] Lustre: 164360:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 5698.286514] Lustre: 164360:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 5698.289448] Lustre: 164360:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 5698.301109] Lustre: 164360:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 5698.308705] Lustre: 164360:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 5700.794609] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 5706.579686] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5709.228635] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5713.531359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5713.562672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5717.431114] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 5722.497956] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5723.680903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5723.684506] Lustre: Skipped 1 previous similar message [ 5723.690308] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5723.770337] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5725.093264] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5725.218183] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5727.467823] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5727.592143] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5729.249326] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5729.256619] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5729.264959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5729.268675] Lustre: Skipped 2 previous similar messages [ 5730.820902] LustreError: 166500:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5730.824406] LustreError: 166500:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5730.948876] Lustre: server umount lustre-OST0000 complete [ 5739.282164] Lustre: server umount lustre-MDT0000 complete [ 5741.331515] LustreError: 164342:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761243177 with bad export cookie 11676478698500309936 [ 5741.333518] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5741.341946] LustreError: 164342:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5750.874897] Lustre: server umount lustre-MDT0001 complete [ 5756.754548] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 14:13:11 (1761243191) [ 5761.752055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5762.157867] LustreError: 167574:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5762.167589] LustreError: 167574:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 5765.429072] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 5771.627678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5774.182757] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 5778.105905] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5781.080373] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5786.740650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5791.497065] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 5796.578276] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5805.517927] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 14:14:00 (1761243240) [ 5808.608735] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5808.612638] LustreError: Skipped 1 previous similar message [ 5808.615586] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5808.623408] Lustre: Skipped 2 previous similar messages [ 5808.631280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5808.633725] Lustre: Skipped 5 previous similar messages [ 5813.875174] LustreError: 170018:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5813.879483] LustreError: 170018:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5813.969923] Lustre: server umount lustre-OST0000 complete [ 5822.699968] Lustre: server umount lustre-MDT0000 complete [ 5825.448645] LustreError: 167554:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761243261 with bad export cookie 11676478698500310860 [ 5825.450295] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5825.454631] LustreError: 167554:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5825.666100] Lustre: server umount lustre-MDT0001 complete [ 5832.026466] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 5833.609706] Key type lgssc unregistered [ 5833.780419] LNet: 170953:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5833.784315] LNetError: 170953:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5833.800725] LNet: Removed LNI 192.168.206.120@tcp [ 5834.271233] Key type .llcrypt unregistered [ 5834.272941] Key type ._llcrypt unregistered [ 5847.383894] Key type ._llcrypt registered [ 5847.387848] Key type .llcrypt registered [ 5847.549949] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 5848.549677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5848.568188] alg: No test for adler32 (adler32-zlib) [ 5849.678156] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 5849.858929] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 5851.631166] Key type lgssc registered [ 5853.025779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5861.884668] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5861.927901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5877.603892] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5882.719229] LustreError: 172156:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.120@tcp: failed processing log, type 4: rc = -110 [ 5887.839516] LustreError: 172156:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5887.845554] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 5887.852320] LustreError: MGC192.168.206.120@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 5887.862895] LustreError: 172156:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-OST0000: -22 [ 5887.876726] LustreError: 172156:0:(tgt_mount.c:2482:server_fill_super()) Unable to start targets: -22 [ 5887.891518] LustreError: 172156:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 5887.911925] LustreError: 172156:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 5888.055725] Lustre: server umount lustre-OST0000 complete [ 5888.057665] LustreError: 172156:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5895.962853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5897.676317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5901.827470] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 5906.913879] LustreError: 172658:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5906.935556] LustreError: 172658:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5909.822703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5910.167595] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5913.216875] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 5917.588738] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5920.418927] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5926.172786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5926.355696] LustreError: 174251:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5926.360851] Lustre: 174251:0:(mgc_request_server.c:712:mgc_llog_local_copy()) MGC192.168.206.120@tcp: invalid local config log lustre-OST0000: rc = -22 [ 5926.473653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5931.145373] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 5935.713978] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5940.194306] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5940.205779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5940.204402] Lustre: Skipped 1 previous similar message [ 5945.315847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5945.318664] Lustre: Skipped 3 previous similar messages [ 5950.434800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5950.437905] Lustre: Skipped 2 previous similar messages [ 5951.969832] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5952.046487] LustreError: 174903:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5952.049163] LustreError: 174903:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5952.093991] Lustre: server umount lustre-MDT0000 complete [ 5954.634956] LustreError: 174904:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761243390 with bad export cookie 8208858731173170473 [ 5954.638229] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5954.676078] LustreError: 174904:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5954.804894] LustreError: 175106:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5954.809369] LustreError: 175106:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5955.011847] Lustre: server umount lustre-MDT0001 complete [ 5966.778530] LustreError: 175307:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5966.799772] LustreError: 175307:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5967.062734] Lustre: server umount lustre-OST0000 complete [ 5975.485827] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [ 5984.338673] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 5992.594546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5997.285125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6002.202127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6006.634367] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6011.799843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6011.849608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6012.067551] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6012.113645] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6012.160611] Lustre: lustre-MDT0000: new disk, initializing [ 6012.235222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6012.250415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6015.272559] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6022.840449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6022.894609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6022.960613] Lustre: 177646:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6023.013740] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6023.016703] Lustre: Skipped 1 previous similar message [ 6023.079053] Lustre: lustre-MDT0001: new disk, initializing [ 6023.180677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6023.223708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6023.231957] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6023.339897] Lustre: 178448:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 6023.349245] Lustre: 178448:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 6023.361624] Lustre: 178448:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 6023.377955] Lustre: 178448:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 6023.391453] Lustre: 178448:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 6023.398020] Lustre: 178448:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 6026.611384] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6031.950154] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6034.250407] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6038.506802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6038.558279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6038.760289] Lustre: lustre-OST0000: new disk, initializing [ 6038.764047] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6038.840550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6042.966421] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6044.716052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6044.723658] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6044.797623] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6048.716686] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6051.735257] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6051.940921] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6055.448262] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6055.719031] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6060.001845] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6060.003803] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6060.005694] LustreError: Skipped 1 previous similar message [ 6060.019318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6060.024150] Lustre: Skipped 2 previous similar messages [ 6060.042425] Lustre: Skipped 1 previous similar message [ 6063.486821] LustreError: 180585:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6063.497539] LustreError: 180585:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6063.668888] Lustre: server umount lustre-OST0000 complete [ 6068.193470] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6068.202792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6068.208555] Lustre: Skipped 1 previous similar message [ 6068.217439] Lustre: Skipped 2 previous similar messages [ 6072.616592] LustreError: 180786:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6072.621452] LustreError: 180786:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6072.703979] Lustre: server umount lustre-MDT0000 complete [ 6073.314976] LustreError: 177656:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6073.325568] LustreError: 177656:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6076.041904] LustreError: 177638:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761243512 with bad export cookie 8208858731173171187 [ 6076.050302] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6076.053143] LustreError: 177638:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6076.415192] Lustre: server umount lustre-MDT0001 complete [ 6084.311132] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 14:18:38 (1761243518) [ 6100.270409] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 6103.020499] Key type lgssc unregistered [ 6103.257290] LNet: 182171:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6103.269356] LNetError: 182171:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6103.279611] LNet: Removed LNI 192.168.206.120@tcp [ 6104.048195] Key type .llcrypt unregistered [ 6104.062098] Key type ._llcrypt unregistered [ 6120.828681] Key type ._llcrypt registered [ 6120.831155] Key type .llcrypt registered [ 6120.922741] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 6121.768584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6121.796086] alg: No test for adler32 (adler32-zlib) [ 6122.899593] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 6123.167464] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 6124.855388] Key type lgssc registered [ 6125.927675] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6132.805952] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6132.821753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6134.354775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6137.923716] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6144.485583] LustreError: 183372:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6144.512285] LustreError: 183372:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6145.800390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6146.089793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6149.934831] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6155.367063] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6158.946324] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6165.886844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6166.236911] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6166.238789] LustreError: 185014:0:(llog.c:1657:llog_backup()) MGC192.168.206.120@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6166.258675] Lustre: 185014:0:(mgc_request_server.c:709:mgc_llog_local_copy()) MGC192.168.206.120@tcp: can't backup local config lustre-OST0000: rc = -28 [ 6166.414866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6171.763334] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6176.379253] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6181.858479] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6181.871545] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6181.873249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6181.885838] Lustre: Skipped 1 previous similar message [ 6184.546195] LustreError: 185666:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6184.649405] Lustre: server umount lustre-MDT0000 complete [ 6186.977282] LustreError: 183371:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6188.525772] LustreError: 183353:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761243624 with bad export cookie 11883114820337426497 [ 6188.535751] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6188.543795] LustreError: 183353:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6188.756148] LustreError: 185870:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6188.759133] LustreError: 185870:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6189.021801] Lustre: server umount lustre-MDT0001 complete [ 6203.142457] LustreError: 186071:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6203.147205] LustreError: 186071:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6203.334598] Lustre: server umount lustre-OST0000 complete [ 6211.910215] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [ 6221.543363] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 6230.796872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6236.579967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6242.416915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6248.121223] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6254.036927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6254.106190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6254.264979] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6254.303348] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6254.355202] Lustre: lustre-MDT0000: new disk, initializing [ 6254.443927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6254.463901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6257.339535] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6266.023908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6266.091258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6266.179052] Lustre: 188410:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6266.218805] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6266.220228] Lustre: Skipped 1 previous similar message [ 6266.222250] LustreError: 189178:0:(llog.c:1657:llog_backup()) MGC192.168.206.120@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 6266.225410] LustreError: 189178:0:(llog.c:1657:llog_backup()) Skipped 1 previous similar message [ 6266.229105] Lustre: 189178:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.206.120@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 6266.244429] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6266.246934] Lustre: Skipped 1 previous similar message [ 6266.291239] Lustre: lustre-MDT0001: new disk, initializing [ 6266.378952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6266.410873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6266.417434] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6266.476924] Lustre: 189210:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 6266.486421] Lustre: 189210:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 6266.493610] Lustre: 189210:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 6266.500985] Lustre: 189210:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 6266.507947] Lustre: 189210:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 6266.513819] Lustre: 189210:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 6269.551654] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6275.946931] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6278.972885] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6284.475754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6284.552562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6284.762842] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6284.764765] Lustre: Skipped 1 previous similar message [ 6284.775776] LustreError: 190135:0:(llog.c:1657:llog_backup()) MGC192.168.206.120@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6284.780899] LustreError: 190135:0:(llog.c:1657:llog_backup()) Skipped 1 previous similar message [ 6284.786592] Lustre: 190135:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.206.120@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 6284.805545] Lustre: 190135:0:(mgc_request_server.c:732:mgc_llog_local_copy()) Skipped 1 previous similar message [ 6284.841800] Lustre: lustre-OST0000: new disk, initializing [ 6284.859021] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6285.007459] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6290.626768] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6291.497481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6291.513596] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6291.650704] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6297.307315] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6300.675993] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6300.897435] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6304.147598] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6304.359273] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6307.301256] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6307.307829] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6307.332718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6307.334340] Lustre: Skipped 2 previous similar messages [ 6312.180149] LustreError: 191352:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6312.194749] LustreError: 191352:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6312.377321] Lustre: server umount lustre-OST0000 complete [ 6317.537473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6317.537473] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6317.537481] Lustre: Skipped 1 previous similar message [ 6317.537969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6317.541748] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6317.579184] Lustre: Skipped 2 previous similar messages [ 6321.852559] LustreError: 191554:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6321.861386] LustreError: 191554:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6321.960208] Lustre: server umount lustre-MDT0000 complete [ 6322.656219] LustreError: 188422:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6322.675339] LustreError: 188422:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6324.969412] LustreError: 188401:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761243761 with bad export cookie 11883114820337427204 [ 6324.971488] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6324.983022] LustreError: 188401:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6325.425960] Lustre: server umount lustre-MDT0001 complete [ 6332.797426] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 14:22:47 (1761243767) [ 6346.106731] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 6354.982518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6355.509827] LustreError: 193123:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6355.577631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6359.039066] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6361.057437] LustreError: 193123:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6365.878665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6369.304837] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6372.017412] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6377.591363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6382.949814] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6390.084484] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6390.151737] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6390.285741] Lustre: lustre-OST0001: new disk, initializing [ 6390.289601] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6390.369777] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6390.372823] Lustre: Skipped 2 previous similar messages [ 6395.264691] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6399.601127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6399.609414] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6399.714679] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6403.828482] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6408.180919] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6413.585129] Lustre: 193119:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 525, rollback = 2 [ 6413.598400] Lustre: 193119:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 1 previous similar message [ 6413.608133] Lustre: 193119:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 6413.611712] Lustre: 193119:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6413.617402] Lustre: 193119:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 6/525/0 [ 6413.625731] Lustre: 193119:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6413.637955] Lustre: 193119:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/100/0, punch: 0/0/0, quota 1/3/0 [ 6413.643489] Lustre: 193119:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6413.650281] Lustre: 193119:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 6413.659595] Lustre: 193119:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6413.663404] Lustre: 193119:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 6413.671606] Lustre: 193119:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6415.036459] Lustre: DEBUG MARKER: ADD OST3 [ 6419.888115] loop: module loaded [ 6419.974333] loop0: detected capacity change from 0 to 204800000 [ 6420.204765] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6420.215483] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6420.224953] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6420.317647] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6425.739241] loop0: detected capacity change from 0 to 204800000 [ 6426.734523] Lustre: DEBUG MARKER: STOP OST3 [ 6431.155671] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6431.258418] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6431.470889] Lustre: lustre-OST0003: new disk, initializing [ 6431.473566] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 6433.061572] Lustre: *** cfs_fail_loc=2109, val=2*** [ 6433.219613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 6433.231405] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 6433.326392] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 6435.114506] LustreError: 197550:0:(osp_precreate.c:1280:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 6435.281099] Lustre: Failing over lustre-OST0003 [ 6435.350807] LustreError: 197774:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 6435.359587] LustreError: 197774:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6435.500431] Lustre: server umount lustre-OST0003 complete [ 6437.039572] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6437.980350] LustreError: 194579:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.206.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6437.993238] LustreError: 194579:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6438.381696] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6438.383211] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6438.403672] Lustre: Skipped 1 previous similar message [ 6444.290377] Lustre: DEBUG MARKER: START OST3 again [ 6449.910629] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6450.169300] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 6451.588035] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 6451.649863] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 6451.653111] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6454.482547] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6474.106396] Lustre: Failing over lustre-OST0003 [ 6474.145657] LustreError: 199728:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 6474.152291] LustreError: 199728:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6474.206179] Lustre: server umount lustre-OST0003 complete [ 6475.754551] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6475.766142] LustreError: 194577:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6475.769640] Lustre: Skipped 1 previous similar message [ 6475.785065] LustreError: 194577:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 6491.110542] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6491.115882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6491.116658] Lustre: Skipped 3 previous similar messages [ 6491.121379] Lustre: Skipped 1 previous similar message [ 6496.231516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6496.245591] Lustre: Skipped 3 previous similar messages [ 6496.719655] Lustre: server umount lustre-MDT0000 complete [ 6499.493718] LustreError: 193104:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761243935 with bad export cookie 11883114820337428135 [ 6499.501898] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6499.515385] LustreError: 193104:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6504.423482] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6504.425955] Lustre: Skipped 3 previous similar messages [ 6505.987950] Lustre: server umount lustre-MDT0001 complete [ 6509.610134] LustreError: 200403:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6509.612818] LustreError: 200403:0:(obd_class.h:479:obd_check_dev()) Skipped 23 previous similar messages [ 6509.641897] Lustre: server umount lustre-OST0000 complete [ 6512.909690] Lustre: server umount lustre-OST0001 complete [ 6518.673109] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [ 6526.732650] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 6535.754903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6540.954262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6546.254198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6552.034351] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6557.219474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6557.278804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6557.452806] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6557.477728] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6557.522689] Lustre: lustre-MDT0000: new disk, initializing [ 6557.590119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6557.593629] Lustre: Skipped 2 previous similar messages [ 6557.603567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6560.218473] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6568.145051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6568.187387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6568.255105] Lustre: 202790:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6568.451131] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6568.528764] Lustre: 203587:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 6568.538523] Lustre: 203587:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 2 previous similar messages [ 6568.544025] Lustre: 203587:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 6568.553936] Lustre: 203587:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 2 previous similar messages [ 6568.567293] Lustre: 203587:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 6568.573158] Lustre: 203587:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 2 previous similar messages [ 6568.578434] Lustre: 203587:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 6568.585204] Lustre: 203587:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 2 previous similar messages [ 6568.592387] Lustre: 203587:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 6568.598656] Lustre: 203587:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 2 previous similar messages [ 6568.603238] Lustre: 203587:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 6568.606751] Lustre: 203587:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 2 previous similar messages [ 6571.106777] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6577.124619] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6580.070551] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6585.560825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6585.619492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6585.873668] Lustre: lustre-OST0000: new disk, initializing [ 6585.881155] Lustre: Skipped 1 previous similar message [ 6585.889850] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6585.897858] Lustre: Skipped 2 previous similar messages [ 6587.616478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6587.619092] Lustre: Skipped 1 previous similar message [ 6587.627613] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6587.772351] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6592.015273] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6598.543345] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6601.892869] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6602.085959] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6605.451382] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6605.811785] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6608.370421] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6608.375400] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6608.378191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6608.382165] Lustre: Skipped 3 previous similar messages [ 6613.580302] LustreError: 205729:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6613.587549] LustreError: 205729:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 6613.773648] Lustre: server umount lustre-OST0000 complete [ 6623.687481] Lustre: server umount lustre-MDT0000 complete [ 6623.712964] LustreError: 202797:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6623.734427] LustreError: 202797:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 6627.176917] LustreError: 205932:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761244063 with bad export cookie 11883114820337430431 [ 6627.192684] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6627.688123] Lustre: server umount lustre-MDT0001 complete [ 6630.607906] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 14:27:45 (1761244065) [ 6642.315539] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [ 6648.856972] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 6656.223242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6662.024137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6667.869065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6673.878783] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6681.064805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6681.151949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6681.344624] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6681.393601] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6681.476955] Lustre: lustre-MDT0000: new disk, initializing [ 6681.595903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6684.805673] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6693.725545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6693.787052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6693.853421] Lustre: 208979:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6694.049224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6694.053953] Lustre: Skipped 3 previous similar messages [ 6694.087891] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6694.170974] Lustre: 209774:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 6694.176064] Lustre: 209774:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 1 previous similar message [ 6694.181811] Lustre: 209774:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 6694.188977] Lustre: 209774:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6694.192819] Lustre: 209774:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 6694.198355] Lustre: 209774:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6694.202564] Lustre: 209774:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 6694.208023] Lustre: 209774:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6694.213126] Lustre: 209774:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 6694.216661] Lustre: 209774:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6694.221596] Lustre: 209774:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 6694.228010] Lustre: 209774:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6697.073918] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6704.199459] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6707.389395] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6713.809855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6713.857332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6715.920048] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6719.278933] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6725.593224] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6729.377922] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6729.564726] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6732.904961] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6733.174704] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6736.353687] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6736.363616] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6736.379135] LustreError: Skipped 2 previous similar messages [ 6736.380148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6736.380153] Lustre: Skipped 5 previous similar messages [ 6736.417815] Lustre: Skipped 3 previous similar messages [ 6741.267311] Lustre: server umount lustre-OST0000 complete [ 6750.729725] LustreError: 212121:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6750.743817] LustreError: 212121:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 6750.812817] Lustre: server umount lustre-MDT0000 complete [ 6751.713043] LustreError: 208990:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6751.719063] LustreError: 208990:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6754.170391] LustreError: 208971:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761244190 with bad export cookie 11883114820337431355 [ 6754.178165] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6754.184796] LustreError: 208971:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6754.689646] Lustre: server umount lustre-MDT0001 complete [ 6761.897687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6765.449539] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6772.043128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6775.581892] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6779.958114] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6782.761825] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6787.500298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6792.278973] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 6796.979322] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6817.763405] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6817.783378] LustreError: Skipped 1 previous similar message [ 6817.791504] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6817.819992] Lustre: Skipped 2 previous similar messages [ 6817.837499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6817.855183] Lustre: Skipped 5 previous similar messages [ 6822.361558] Lustre: server umount lustre-OST0000 complete [ 6832.508661] Lustre: server umount lustre-MDT0000 complete [ 6836.062166] LustreError: 212778:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761244272 with bad export cookie 11883114820337432272 [ 6836.071118] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6836.072147] LustreError: 212778:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6836.575229] Lustre: server umount lustre-MDT0001 complete [ 6934.526291] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 6937.687720] Key type lgssc unregistered [ 6937.977552] LNet: 216132:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6937.985045] LNetError: 216132:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6938.006883] LNet: Removed LNI 192.168.206.120@tcp [ 6939.059272] Key type .llcrypt unregistered [ 6939.062928] Key type ._llcrypt unregistered [ 6961.763556] Key type ._llcrypt registered [ 6961.765475] Key type .llcrypt registered [ 6961.853054] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [ 6974.786212] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 6975.553587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6975.665064] alg: No test for adler32 (adler32-zlib) [ 6976.846984] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 6977.125854] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 6978.793588] Key type lgssc registered [ 6979.719454] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6986.645451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6991.026074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6995.421935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7000.561762] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7005.725502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7005.757441] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7005.768163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7006.924345] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7006.961781] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7007.021861] Lustre: lustre-MDT0000: new disk, initializing [ 7007.107958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7007.126313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7009.973115] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 7018.225884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7018.275417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7018.343752] Lustre: 219457:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7018.388567] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7018.392196] Lustre: Skipped 1 previous similar message [ 7018.471746] Lustre: lustre-MDT0001: new disk, initializing [ 7018.544901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7018.587854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7018.599763] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7018.669314] Lustre: 220253:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 7018.679693] Lustre: 220253:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 7018.695373] Lustre: 220253:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 7018.703938] Lustre: 220253:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 7018.707797] Lustre: 220253:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 7018.716401] Lustre: 220253:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 7021.593062] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 7027.259800] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7030.115372] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7035.138301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7035.179777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7035.319884] Lustre: lustre-OST0000: new disk, initializing [ 7035.322454] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7035.416126] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7039.913925] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 7044.697949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7044.702607] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7044.823742] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7046.212085] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7049.723263] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7049.907905] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7053.335278] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7053.532657] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7055.333751] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7055.334966] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7055.346982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7055.353107] LustreError: Skipped 1 previous similar message [ 7060.453077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7060.460255] Lustre: Skipped 2 previous similar messages [ 7065.573440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7065.579343] Lustre: Skipped 1 previous similar message [ 7069.151116] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7069.188304] LustreError: 222400:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7069.372107] Lustre: server umount lustre-OST0000 complete [ 7074.784446] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7074.794507] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7074.802218] Lustre: Skipped 1 previous similar message [ 7074.810418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7075.808940] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7079.186115] LustreError: 222602:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7079.188319] LustreError: 222602:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7079.244220] Lustre: server umount lustre-MDT0000 complete [ 7081.504070] LustreError: 219450:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761244517 with bad export cookie 1122745434620450725 [ 7081.507477] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7081.517343] LustreError: 219450:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7081.597913] LustreError: 222804:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7081.600530] LustreError: 222804:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7081.736301] Lustre: server umount lustre-MDT0001 complete [ 7094.951370] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 7102.134263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7102.472220] LustreError: 223978:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7102.483515] LustreError: 223978:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7102.517258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7105.452898] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 7107.554819] LustreError: 223978:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7111.934344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7112.188092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7114.799147] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 7116.977545] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7121.157308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7121.466156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7125.640187] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 7133.093519] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7133.147550] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7133.265839] Lustre: lustre-OST0001: new disk, initializing [ 7133.270358] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7137.894622] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 7141.023324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7141.029505] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7141.114239] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7147.003289] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7159.509816] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7175.532027] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 14:36:50 (1761244610) [ 7177.190161] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7177.192053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7177.195093] Lustre: Skipped 2 previous similar messages [ 7177.197891] Lustre: Skipped 3 previous similar messages [ 7183.173027] LustreError: 227378:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 7183.184712] LustreError: 227378:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7183.277788] Lustre: server umount lustre-MDT0000 complete [ 7186.030450] LustreError: 223957:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761244622 with bad export cookie 1122745434620451649 [ 7186.031582] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7186.035790] LustreError: 223957:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7186.404165] Lustre: server umount lustre-MDT0001 complete [ 7199.176559] LustreError: 227780:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7199.184599] LustreError: 227780:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 7199.221270] Lustre: server umount lustre-OST0000 complete [ 7212.032191] LustreError: 227982:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7212.039489] LustreError: 227982:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7212.168194] Lustre: server umount lustre-OST0001 complete [ 7217.561580] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [ 7226.806564] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 7238.044192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7244.658915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7250.453859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7255.835096] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7261.373154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7261.481471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7261.668298] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7261.700185] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7261.759244] Lustre: lustre-MDT0000: new disk, initializing [ 7261.877493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7261.884307] Lustre: Skipped 1 previous similar message [ 7261.907561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7264.990711] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 7273.431285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7273.488932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7273.551379] Lustre: 230171:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7273.591860] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7273.594942] Lustre: Skipped 1 previous similar message [ 7273.670672] Lustre: lustre-MDT0001: new disk, initializing [ 7273.859529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7273.871896] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7273.924823] Lustre: 230969:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 7273.930051] Lustre: 230969:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 1 previous similar message [ 7273.933476] Lustre: 230969:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 7273.939259] Lustre: 230969:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 1 previous similar message [ 7273.948591] Lustre: 230969:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 7273.953780] Lustre: 230969:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 7273.957792] Lustre: 230969:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 7273.968816] Lustre: 230969:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 7273.972752] Lustre: 230969:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 7273.976604] Lustre: 230969:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 7273.979780] Lustre: 230969:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 7273.985161] Lustre: 230969:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 7276.899477] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 7283.334450] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7286.053322] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7290.575309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7290.624219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7290.826720] Lustre: lustre-OST0000: new disk, initializing [ 7290.829501] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7292.263751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7292.267754] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7292.342142] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7295.098913] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 7300.409447] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7303.266900] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7303.441578] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7306.135468] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7306.258460] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7307.745098] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7307.746095] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7307.755559] LustreError: Skipped 1 previous similar message [ 7307.757479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7307.762583] Lustre: Skipped 2 previous similar messages [ 7307.771962] Lustre: Skipped 5 previous similar messages [ 7313.459124] LustreError: 233112:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7313.462270] LustreError: 233112:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7313.584867] Lustre: server umount lustre-OST0000 complete [ 7317.984691] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7318.001325] Lustre: Skipped 1 previous similar message [ 7322.767069] Lustre: server umount lustre-MDT0000 complete [ 7323.105688] LustreError: 230177:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7323.125772] LustreError: 230177:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7326.074247] LustreError: 230953:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761244762 with bad export cookie 1122745434620453791 [ 7326.077622] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7326.089869] LustreError: 230953:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7326.464468] Lustre: server umount lustre-MDT0001 complete [ 7334.067579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7334.600402] LustreError: 233992:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7334.712912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7334.719254] Lustre: Skipped 2 previous similar messages [ 7338.007558] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 7340.003654] LustreError: 233992:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7345.574357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7349.134381] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 7353.391179] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7356.371839] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7362.501668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7368.655217] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 7374.013153] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7523.811194] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7523.815017] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7523.836184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7523.845382] Lustre: Skipped 1 previous similar message [ 7523.858349] Lustre: Skipped 5 previous similar messages [ 7527.233478] LustreError: 236280:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7527.236346] LustreError: 236280:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 7527.473130] Lustre: server umount lustre-OST0000 complete [ 7537.873438] Lustre: server umount lustre-MDT0000 complete [ 7539.171847] LustreError: 233988:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7539.184488] LustreError: 233988:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7540.652544] LustreError: 236485:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761244976 with bad export cookie 1122745434620454708 [ 7540.654923] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7540.659417] LustreError: 236485:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7540.871175] Lustre: server umount lustre-MDT0001 complete [ 7638.393866] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 7640.379983] Key type lgssc unregistered [ 7640.598960] LNet: 237249:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7640.605375] LNetError: 237249:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7640.619099] LNet: Removed LNI 192.168.206.120@tcp [ 7641.177188] Key type .llcrypt unregistered [ 7641.180804] Key type ._llcrypt unregistered [ 7659.339854] Key type ._llcrypt registered [ 7659.341372] Key type .llcrypt registered [ 7659.409405] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [ 7670.758243] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 7671.529678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7671.541468] alg: No test for adler32 (adler32-zlib) [ 7672.602242] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 7672.781159] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 7674.407202] Key type lgssc registered [ 7675.295552] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7681.583264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7685.913759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7690.614607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7695.396109] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7706.330183] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 7712.326574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7712.385349] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7712.404477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7713.582342] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7713.613780] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7713.666908] Lustre: lustre-MDT0000: new disk, initializing [ 7713.777828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7713.807428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7716.597881] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 7724.503321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7724.545488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7724.618938] Lustre: 241256:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7724.672145] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7724.674695] Lustre: Skipped 1 previous similar message [ 7724.746118] Lustre: lustre-MDT0001: new disk, initializing [ 7724.844453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7724.879748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7724.889094] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7724.997606] Lustre: 242050:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 7725.009652] Lustre: 242050:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 7725.020807] Lustre: 242050:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 7725.032372] Lustre: 242050:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 7725.042215] Lustre: 242050:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 7725.053096] Lustre: 242050:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 7727.953622] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 7732.005217] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7737.737744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7737.846243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7738.184421] Lustre: lustre-OST0000: new disk, initializing [ 7738.194225] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 7738.199227] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7738.305153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7743.462730] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 7743.552552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7743.562805] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7743.607566] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 7752.885276] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7752.944967] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7753.067882] Lustre: lustre-OST0001: new disk, initializing [ 7753.073858] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7753.186387] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7758.300180] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 7759.971893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7759.985938] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7760.053787] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7768.138833] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7772.868227] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7786.464290] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7786.469175] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7786.477202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7791.073392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7791.077216] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7791.092097] Lustre: Skipped 1 previous similar message [ 7792.622063] LustreError: 244781:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 7792.691892] Lustre: server umount lustre-MDT0000 complete [ 7795.931222] LustreError: 241249:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761245232 with bad export cookie 16439216709617164759 [ 7795.932696] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7795.937214] LustreError: 241249:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7796.051520] LustreError: 244982:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 7796.057130] LustreError: 244982:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7796.272313] Lustre: server umount lustre-MDT0001 complete [ 7810.006557] LustreError: 245183:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 7810.013311] LustreError: 245183:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7810.068632] Lustre: server umount lustre-OST0000 complete [ 7812.583933] Lustre: 238767:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761245232/real 1761245232] req@ffff932b6fb0b100 x1846799351493888/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761245248 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7812.616994] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7812.634814] Lustre: Skipped 1 previous similar message [ 7813.342464] LustreError: 245385:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7813.345313] LustreError: 245385:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7813.594256] Lustre: server umount lustre-OST0001 complete [ 7826.053140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7833.214864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7834.020276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7841.301339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7848.976962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7849.764710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7865.927200] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 7875.051673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7875.067474] Lustre: lustre-MDT0000: reset Object Index mappings [ 7875.502149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7878.968222] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 7880.674081] LustreError: 248696:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7880.706765] LustreError: 248696:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7884.780428] LustreError: 248695:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7886.433716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7886.440713] Lustre: lustre-MDT0001: reset Object Index mappings [ 7886.885864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7890.230130] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 7893.499571] Lustre: 249808:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7899.755304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7901.103337] LustreError: 250165:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7901.165987] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 7905.617995] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 7906.287161] LustreError: 250165:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7911.393558] LustreError: 250246:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7912.593702] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7912.825310] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7912.828609] Lustre: Skipped 1 previous similar message [ 7914.848761] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 7918.149871] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 7924.064275] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7926.571041] Lustre: 251654:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7938.017825] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7938.027629] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7938.035519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7938.039233] Lustre: Skipped 3 previous similar messages [ 7940.803625] LustreError: 251759:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 7940.815430] LustreError: 251759:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7940.900182] Lustre: server umount lustre-MDT0000 complete [ 7943.618895] LustreError: 248677:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761245379 with bad export cookie 16439216709617171591 [ 7943.624598] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7943.634815] LustreError: 248677:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7943.673228] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7943.675848] LustreError: 250551:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7943.682961] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7943.682970] Lustre: Skipped 3 previous similar messages [ 7943.687149] Lustre: Skipped 4 previous similar messages [ 7943.705014] LustreError: 250551:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7943.935525] Lustre: server umount lustre-MDT0001 complete [ 7952.823535] LustreError: 252161:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 7952.835310] LustreError: 252161:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 7952.906549] Lustre: server umount lustre-OST0000 complete [ 7956.000409] Lustre: server umount lustre-OST0001 complete [ 7970.925986] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 14:50:05 (1761245405) [ 7985.549417] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [ 7994.122626] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 8002.608669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8009.281050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8015.727672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8022.476125] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8028.371960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8028.454850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8028.700352] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8028.738440] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8028.779255] Lustre: lustre-MDT0000: new disk, initializing [ 8028.884533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8028.902789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8031.969577] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 8040.656846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8040.725725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8040.810637] Lustre: 255890:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8040.874639] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8040.877507] Lustre: Skipped 1 previous similar message [ 8040.925061] Lustre: lustre-MDT0001: new disk, initializing [ 8041.077188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8041.084334] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8041.198388] Lustre: 256685:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 8041.230226] Lustre: 256685:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 1 previous similar message [ 8041.246471] Lustre: 256685:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 8041.249901] Lustre: 256685:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 1 previous similar message [ 8041.271194] Lustre: 256685:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 8041.280605] Lustre: 256685:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 8041.290653] Lustre: 256685:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 8041.307832] Lustre: 256685:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 8041.311430] Lustre: 256685:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 8041.317270] Lustre: 256685:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 8041.334262] Lustre: 256685:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 8041.336662] Lustre: 256685:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 8044.619698] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 8051.474873] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8054.604909] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8060.399663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8060.463701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8060.690747] Lustre: lustre-OST0000: new disk, initializing [ 8060.695816] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8062.739169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8062.748690] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8062.907343] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8066.103926] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 8073.796376] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8078.413155] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8078.749887] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8083.554811] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8083.848653] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8088.550524] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8088.554172] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8088.559670] Lustre: Skipped 1 previous similar message [ 8088.562976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8088.565909] Lustre: Skipped 2 previous similar messages [ 8100.319397] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8100.374344] LustreError: 258827:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8100.388522] LustreError: 258827:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 8100.628994] Lustre: server umount lustre-OST0000 complete [ 8109.025748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8109.038221] Lustre: Skipped 7 previous similar messages [ 8118.239119] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8118.515063] Lustre: server umount lustre-MDT0000 complete [ 8119.270334] LustreError: 256689:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8119.295300] LustreError: 256689:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8122.735795] LustreError: 255882:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761245559 with bad export cookie 16439216709617173684 [ 8122.736835] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8122.758731] LustreError: 255882:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8123.122649] Lustre: server umount lustre-MDT0001 complete [ 8141.665302] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 8152.644515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8153.220665] LustreError: 260410:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8153.267615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8153.275287] Lustre: Skipped 2 previous similar messages [ 8157.138869] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 8165.190151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8168.960396] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 8171.486867] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8177.006278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8182.516336] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 8190.766253] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8190.869923] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8191.045155] Lustre: lustre-OST0001: new disk, initializing [ 8191.047720] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8192.808560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8192.830181] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8192.922452] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8196.602590] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 8204.678887] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8207.417772] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8216.509360] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 8216.727160] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 8222.179530] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8222.189494] LustreError: Skipped 1 previous similar message [ 8222.198592] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8222.225598] Lustre: Skipped 2 previous similar messages [ 8222.231877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8222.242160] Lustre: Skipped 2 previous similar messages [ 8223.910874] LustreError: 263918:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 8223.919814] LustreError: 263918:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 8224.146832] Lustre: server umount lustre-MDT0000 complete [ 8227.846282] LustreError: 260389:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761245664 with bad export cookie 16439216709617174608 [ 8227.853268] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8227.857897] LustreError: 260389:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8228.352974] Lustre: server umount lustre-MDT0001 complete [ 8242.449151] Lustre: server umount lustre-OST0000 complete [ 8255.906383] Lustre: server umount lustre-OST0001 complete [ 8269.141422] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 8278.640098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8279.199706] LustreError: 265701:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8279.222032] LustreError: 265701:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8283.359719] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 8290.757257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8291.191140] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8291.193368] Lustre: Skipped 4 previous similar messages [ 8294.949739] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 8297.585037] Lustre: 266811:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8303.352608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8304.763846] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 8308.868323] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 8316.572827] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8318.121728] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 8321.857406] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 8328.530781] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8332.017296] Lustre: 268669:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8342.437679] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 8342.473517] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 8350.798735] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 14:56:25 (1761245785) [ 8368.103668] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8368.105707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8368.113071] Lustre: Skipped 6 previous similar messages [ 8368.113366] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8368.118774] Lustre: Skipped 7 previous similar messages [ 8370.156411] LustreError: 269375:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 8370.161938] LustreError: 269375:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 8370.214746] Lustre: server umount lustre-MDT0000 complete [ 8373.089797] LustreError: 265682:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761245809 with bad export cookie 16439216709617180887 [ 8373.092310] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8373.126491] LustreError: 265682:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8379.643621] Lustre: server umount lustre-MDT0001 complete [ 8397.279138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8397.380567] Lustre: server umount lustre-OST0000 complete [ 8400.662178] Lustre: server umount lustre-OST0001 complete [ 8406.047469] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [ 8414.996133] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 8423.114141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8429.360811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8434.689367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8440.945451] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8458.463606] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 8468.246700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8468.327803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8468.575880] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8468.615265] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8468.745216] Lustre: lustre-MDT0000: new disk, initializing [ 8468.881953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8472.302803] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 8481.382770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8481.451479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8481.564820] Lustre: 272868:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8481.870895] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8481.938923] Lustre: 273663:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 8481.954849] Lustre: 273663:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 1 previous similar message [ 8481.958847] Lustre: 273663:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 8481.962794] Lustre: 273663:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 1 previous similar message [ 8481.970095] Lustre: 273663:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 8481.975171] Lustre: 273663:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 8481.986574] Lustre: 273663:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 8482.005372] Lustre: 273663:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 8482.019952] Lustre: 273663:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 8482.036085] Lustre: 273663:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 8482.039470] Lustre: 273663:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 8482.065089] Lustre: 273663:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 8485.252344] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 8494.886401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8494.928248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8497.163134] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8499.817933] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 8508.600780] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8508.644088] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8510.261302] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8513.416282] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 8521.512531] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8524.379457] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8524.382238] Lustre: Skipped 1 previous similar message [ 8538.840328] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 14:59:33 (1761245973) [ 8541.155292] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8541.158681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8541.161520] Lustre: Skipped 3 previous similar messages [ 8541.164491] Lustre: Skipped 4 previous similar messages [ 8546.349414] LustreError: 276478:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 8546.358221] LustreError: 276478:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 8546.477836] Lustre: server umount lustre-MDT0000 complete [ 8549.376538] LustreError: 272860:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761245985 with bad export cookie 16439216709617188762 [ 8549.380373] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8549.387907] LustreError: 272860:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8549.645246] Lustre: server umount lustre-MDT0001 complete [ 8562.777284] Lustre: server umount lustre-OST0000 complete [ 8576.924871] Lustre: server umount lustre-OST0001 complete [ 8593.824890] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [ 8602.770523] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 8610.823129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8616.350522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8621.230298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8627.268856] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8634.372619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8634.437824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8634.605335] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8634.642645] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8634.649872] Lustre: Skipped 4 previous similar messages [ 8634.729444] Lustre: lustre-MDT0000: new disk, initializing [ 8634.732387] Lustre: Skipped 3 previous similar messages [ 8634.815771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8634.818025] Lustre: Skipped 6 previous similar messages [ 8634.850067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8634.864598] Lustre: Skipped 3 previous similar messages [ 8638.515408] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 8646.612909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8646.689382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8646.788521] Lustre: 279941:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8647.022993] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8647.026147] Lustre: Skipped 2 previous similar messages [ 8647.108200] Lustre: 280739:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 8647.124064] Lustre: 280739:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 1 previous similar message [ 8647.131904] Lustre: 280739:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 8647.143162] Lustre: 280739:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 1 previous similar message [ 8647.148947] Lustre: 280739:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 8647.166205] Lustre: 280739:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 8647.170549] Lustre: 280739:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 8647.175550] Lustre: 280739:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 8647.179855] Lustre: 280739:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 8647.183602] Lustre: 280739:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 8647.188858] Lustre: 280739:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 8647.192806] Lustre: 280739:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 8650.299447] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 8656.197397] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8658.865391] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8663.701628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8663.767449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8665.619259] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8669.391343] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 8675.670931] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8677.348022] Lustre: Mounted lustre-client [ 8678.189197] LustreError: 282447:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff932a50567800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 8678.260863] Lustre: Unmounted lustre-client [ 8686.047829] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8686.054062] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8686.072708] Lustre: Skipped 3 previous similar messages [ 8699.871140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8700.055842] Lustre: server umount lustre-OST0000 complete [ 8717.279132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8717.425215] Lustre: server umount lustre-MDT0000 complete [ 8718.307310] LustreError: 279952:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8718.317194] LustreError: 279952:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 8720.647263] LustreError: 279933:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761246156 with bad export cookie 16439216709617190330 [ 8720.651549] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8720.656650] LustreError: 279933:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8721.178821] Lustre: server umount lustre-MDT0001 complete [ 8730.181786] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 8732.938603] Key type lgssc unregistered [ 8737.312479] LNet: 283729:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8737.325363] LNetError: 283729:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8807.981987] LNet: Removed LNI 192.168.206.120@tcp [ 8808.964282] Key type .llcrypt unregistered [ 8808.966294] Key type ._llcrypt unregistered [ 8824.059271] Key type ._llcrypt registered [ 8824.063165] Key type .llcrypt registered [ 8824.188477] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [ 8837.000704] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 8837.657815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8837.684218] alg: No test for adler32 (adler32-zlib) [ 8838.757770] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 8838.965368] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 8840.600149] Key type lgssc registered [ 8841.690192] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8849.443185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8854.995849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8860.339455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8866.128635] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8867.809116] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 15:05:02 (1761246302) [ 8873.422390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8873.462948] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8873.479549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8874.765088] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8874.805884] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8874.890568] Lustre: lustre-MDT0000: new disk, initializing [ 8874.993540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8875.020553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8878.138767] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 8886.589403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8886.650250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8886.732568] Lustre: 286919:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8886.791432] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8886.794769] Lustre: Skipped 1 previous similar message [ 8886.846871] Lustre: lustre-MDT0001: new disk, initializing [ 8886.928425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8886.980155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8886.996148] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8887.082781] Lustre: 287714:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 8887.085469] Lustre: 287714:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 8887.102695] Lustre: 287714:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 8887.109883] Lustre: 287714:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 8887.120187] Lustre: 287714:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 8887.123964] Lustre: 287714:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 8890.402926] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 8896.202494] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8899.070597] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8905.182950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8905.273705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8905.694479] Lustre: lustre-OST0000: new disk, initializing [ 8905.718321] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8905.851555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8912.109806] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_default_debug -1 all [ 8913.474560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8913.481649] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8913.588725] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8919.141887] Lustre: DEBUG MARKER: oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8929.255772] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8929.258900] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8929.295549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8929.310927] Lustre: Skipped 1 previous similar message [ 8934.384088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8934.391268] Lustre: Skipped 1 previous similar message [ 8939.496955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8941.023241] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8941.079436] LustreError: 289361:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8941.512198] Lustre: server umount lustre-OST0000 complete [ 8948.704124] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8948.707701] LustreError: Skipped 1 previous similar message [ 8948.711667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8948.737620] Lustre: Skipped 1 previous similar message [ 8948.745314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8948.753732] Lustre: Skipped 1 previous similar message [ 8949.731181] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8951.799474] LustreError: 289565:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8951.801876] LustreError: 289565:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8951.877609] Lustre: server umount lustre-MDT0000 complete [ 8955.041280] LustreError: 286912:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761246391 with bad export cookie 12416297084145583967 [ 8955.043899] LustreError: MGC192.168.206.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8955.067791] LustreError: 286912:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8955.182446] LustreError: 289768:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8955.189979] LustreError: 289768:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8955.405682] Lustre: server umount lustre-MDT0001 complete [ 8965.495984] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 8968.110888] Key type lgssc unregistered [ 8968.480612] LNet: 290301:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8968.504308] LNetError: 290301:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8969.509685] LNet: Removed LNI 192.168.206.120@tcp [ 8970.204483] Key type .llcrypt unregistered [ 8970.206242] Key type ._llcrypt unregistered [ 9054.944521] Key type ._llcrypt registered [ 9054.952802] Key type .llcrypt registered [ 9055.040813] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 9055.251487] Key type .llcrypt unregistered [ 9055.252840] Key type ._llcrypt unregistered [ 9130.883321] Key type ._llcrypt registered [ 9130.884943] Key type .llcrypt registered [ 9131.002134] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 9131.290422] Key type .llcrypt unregistered [ 9131.291817] Key type ._llcrypt unregistered [ 9158.875236] Key type ._llcrypt registered [ 9158.877239] Key type .llcrypt registered [ 9158.960920] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 9159.187449] Key type .llcrypt unregistered [ 9159.198860] Key type ._llcrypt unregistered [ 9233.011624] Key type ._llcrypt registered [ 9233.012891] Key type .llcrypt registered [ 9233.133818] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 9233.360966] Key type .llcrypt unregistered [ 9233.364252] Key type ._llcrypt unregistered [ 9261.724560] Key type ._llcrypt registered [ 9261.726550] Key type .llcrypt registered [ 9261.836382] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 9262.253658] Key type .llcrypt unregistered [ 9262.255125] Key type ._llcrypt unregistered [ 9291.549930] Key type ._llcrypt registered [ 9291.551363] Key type .llcrypt registered [ 9291.662181] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 9291.903321] Key type .llcrypt unregistered [ 9291.904472] Key type ._llcrypt unregistered [ 9362.299087] Key type ._llcrypt registered [ 9362.300617] Key type .llcrypt registered [ 9362.461606] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 9362.756316] Key type .llcrypt unregistered [ 9362.758279] Key type ._llcrypt unregistered [ 9392.625542] Key type ._llcrypt registered [ 9392.627207] Key type .llcrypt registered [ 9392.751606] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 9393.126292] Key type .llcrypt unregistered [ 9393.127775] Key type ._llcrypt unregistered [ 9421.287766] Key type ._llcrypt registered [ 9421.289204] Key type .llcrypt registered [ 9421.437516] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 9421.742345] Key type .llcrypt unregistered [ 9421.743799] Key type ._llcrypt unregistered [ 9471.772594] Key type ._llcrypt registered [ 9471.781208] Key type .llcrypt registered [ 9471.907987] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing set_hostid [ 9487.512592] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing load_modules_local [ 9488.542312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9488.580142] alg: No test for adler32 (adler32-zlib) [ 9489.573596] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 9489.807273] LNet: Added LNI 192.168.206.120@tcp [8/256/0/180] [ 9491.442253] Key type lgssc registered [ 9492.536467] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9500.865896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9507.113196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9512.304131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9518.224659] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9519.876352] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 9213 sec ========== 15:15:54 (1761246954) [ 9521.234073] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 15:15:56 (1761246956) === [ 9523.991714] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 15:15:58 (1761246958) === [ 9557.574418] Lustre: DEBUG MARKER: oleg620-server.virtnet: executing unload_modules_local [ 9560.304872] Key type lgssc unregistered [ 9560.533259] LNet: 309424:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9560.546660] LNetError: 309424:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9560.559155] LNet: Removed LNI 192.168.206.120@tcp [ 9561.229040] Key type .llcrypt unregistered [ 9561.230566] Key type ._llcrypt unregistered