[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 501886386 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.003374] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.007902] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009015] pid_max: default: 32768 minimum: 301 [ 0.010152] LSM: Security Framework initializing [ 0.011071] Yama: becoming mindful. [ 0.012048] SELinux: Initializing. [ 0.013081] *** VALIDATE selinux *** [ 0.021886] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026873] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028174] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029110] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030117] *** VALIDATE tmpfs *** [ 0.031511] *** VALIDATE proc *** [ 0.033257] *** VALIDATE cgroup *** [ 0.034012] *** VALIDATE cgroup2 *** [ 0.035305] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037158] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039042] Spectre V2 : User space: Vulnerable [ 0.040016] Speculative Store Bypass: Vulnerable [ 0.042931] debug: unmapping init [mem 0xffffffff91c59000-0xffffffff91c60fff] [ 0.044145] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045528] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046023] ... version: 2 [ 0.046606] ... bit width: 48 [ 0.047011] ... generic registers: 4 [ 0.048018] ... value mask: 0000ffffffffffff [ 0.049018] ... max period: 00007fffffffffff [ 0.050018] ... fixed-purpose events: 3 [ 0.050974] ... event mask: 000000070000000f [ 0.051363] rcu: Hierarchical SRCU implementation. [ 0.053629] smp: Bringing up secondary CPUs ... [ 0.054661] x86: Booting SMP configuration: [ 0.055022] .... node #0, CPUs: #1 #2 #3 [ 0.058214] smp: Brought up 1 node, 4 CPUs [ 0.059997] smpboot: Max logical packages: 1 [ 0.061021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.157117] node 0 deferred pages initialised in 95ms [ 0.161013] devtmpfs: initialized [ 0.162349] x86/mm: Memory block size: 128MB [ 0.165807] gcov: version magic: 0x41383552 [ 0.168334] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.172125] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.175365] pinctrl core: initialized pinctrl subsystem [ 0.177206] [ 0.177813] ************************************************************* [ 0.180022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.182018] ** ** [ 0.185017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.187014] ** ** [ 0.189020] ** This means that this kernel is built to expose internal ** [ 0.192018] ** IOMMU data structures, which may compromise security on ** [ 0.194023] ** your system. ** [ 0.196018] ** ** [ 0.198016] ** If you see this message and you are not debugging the ** [ 0.200013] ** kernel, report this immediately to your vendor! ** [ 0.202031] ** ** [ 0.204013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.207014] ************************************************************* [ 0.209827] NET: Registered protocol family 16 [ 0.211325] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.214069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.217078] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.222013] cpuidle: using governor menu [ 0.223464] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.225400] PCI: Using configuration type 1 for base access [ 0.228134] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.236149] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.237022] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.239093] cryptd: max_cpu_qlen set to 1000 [ 0.242264] ACPI: Added _OSI(Module Device) [ 0.243012] ACPI: Added _OSI(Processor Device) [ 0.245011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.246024] ACPI: Added _OSI(Processor Aggregator Device) [ 0.250000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.255143] ACPI: Interpreter enabled [ 0.256045] ACPI: PM: (supports S0 S3 S4 S5) [ 0.257012] ACPI: Using IOAPIC for interrupt routing [ 0.259101] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.261331] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.267938] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.270051] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.272026] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.275069] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.279362] acpiphp: Slot [2] registered [ 0.281118] acpiphp: Slot [5] registered [ 0.282140] acpiphp: Slot [6] registered [ 0.283107] acpiphp: Slot [7] registered [ 0.284119] acpiphp: Slot [8] registered [ 0.285160] acpiphp: Slot [9] registered [ 0.286121] acpiphp: Slot [10] registered [ 0.287139] acpiphp: Slot [3] registered [ 0.289125] acpiphp: Slot [4] registered [ 0.290095] acpiphp: Slot [11] registered [ 0.291091] acpiphp: Slot [12] registered [ 0.292096] acpiphp: Slot [13] registered [ 0.293085] acpiphp: Slot [14] registered [ 0.295127] acpiphp: Slot [15] registered [ 0.296304] acpiphp: Slot [16] registered [ 0.297115] acpiphp: Slot [17] registered [ 0.299129] acpiphp: Slot [18] registered [ 0.300130] acpiphp: Slot [19] registered [ 0.302135] acpiphp: Slot [20] registered [ 0.303122] acpiphp: Slot [21] registered [ 0.305118] acpiphp: Slot [22] registered [ 0.306142] acpiphp: Slot [23] registered [ 0.307127] acpiphp: Slot [24] registered [ 0.309141] acpiphp: Slot [25] registered [ 0.311125] acpiphp: Slot [26] registered [ 0.312108] acpiphp: Slot [27] registered [ 0.314117] acpiphp: Slot [28] registered [ 0.315382] acpiphp: Slot [29] registered [ 0.317120] acpiphp: Slot [30] registered [ 0.319111] acpiphp: Slot [31] registered [ 0.320119] PCI host bridge to bus 0000:00 [ 0.322022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.324046] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.326021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.328040] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.330038] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.333033] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.335207] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.336665] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.338785] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.347743] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.354526] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.357025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.359029] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.361023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.363496] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.367867] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.370048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.372822] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.378028] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.392023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.398021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.404992] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.414022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.423018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.450020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.467267] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.479015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.489017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.519027] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.529235] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.536016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.551017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.573028] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.584957] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.598019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.608016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.632026] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.641879] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.657032] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.668018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.690022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.702064] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.710023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.725019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.747021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.758105] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.761443] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.764406] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.766391] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.769259] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.775102] iommu: Default domain type: Passthrough [ 0.777404] SCSI subsystem initialized [ 0.778107] ACPI: bus type USB registered [ 0.779113] usbcore: registered new interface driver usbfs [ 0.781063] usbcore: registered new interface driver hub [ 0.782074] usbcore: registered new device driver usb [ 0.784147] pps_core: LinuxPPS API ver. 1 registered [ 0.785017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.788078] PTP clock support registered [ 0.791099] EDAC MC: Ver: 3.0.0 [ 0.793126] PCI: Using ACPI for IRQ routing [ 0.795070] NetLabel: Initializing [ 0.796013] NetLabel: domain hash size = 128 [ 0.798160] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.800085] NetLabel: unlabeled traffic allowed by default [ 0.802116] vgaarb: loaded [ 0.803325] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.805017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.813349] clocksource: Switched to clocksource kvm-clock [ 0.923537] VFS: Disk quotas dquot_6.6.0 [ 0.925188] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.927965] *** VALIDATE ramfs *** [ 0.929319] *** VALIDATE hugetlbfs *** [ 0.930995] pnp: PnP ACPI init [ 0.933422] pnp: PnP ACPI: found 6 devices [ 0.952205] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.956101] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.959061] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.961515] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.964529] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.967847] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.970890] NET: Registered protocol family 2 [ 0.973528] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.979507] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.982610] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.988374] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.992165] TCP: Hash tables configured (established 65536 bind 65536) [ 0.995366] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.998913] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.001762] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.004890] NET: Registered protocol family 1 [ 1.008258] RPC: Registered named UNIX socket transport module. [ 1.010486] RPC: Registered udp transport module. [ 1.012120] RPC: Registered tcp transport module. [ 1.013845] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.016223] NET: Registered protocol family 44 [ 1.019585] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.021896] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.024151] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.026673] PCI: CLS 0 bytes, default 64 [ 1.028528] Unpacking initramfs... [ 2.446412] debug: unmapping init [mem 0xffff9aa17cc54000-0xffff9aa17ffbffff] [ 2.450375] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.452683] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.455619] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.939361] Initialise system trusted keyrings [ 2.941182] Key type blacklist registered [ 2.942977] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.952284] zbud: loaded [ 2.955215] *** VALIDATE nfs *** [ 2.956519] *** VALIDATE nfs4 *** [ 2.958173] pstore: using deflate compression [ 2.961817] Platform Keyring initialized [ 3.069867] NET: Registered protocol family 38 [ 3.071935] Key type asymmetric registered [ 3.073705] Asymmetric key parser 'x509' registered [ 3.075787] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.079133] io scheduler mq-deadline registered [ 3.080988] io scheduler kyber registered [ 3.083160] io scheduler bfq registered [ 3.084925] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.088160] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.091278] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.094448] ACPI: Power Button [PWRF] [ 3.099984] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.106892] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.122144] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.130243] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.144852] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.173374] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.205179] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.210474] Non-volatile memory driver v1.3 [ 3.211978] Linux agpgart interface v0.103 [ 3.246060] virtio_blk virtio1: [vda] 146936 512-byte logical blocks (75.2 MB/71.7 MiB) [ 3.248864] vda: detected capacity change from 0 to 75231232 [ 3.267402] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.270059] vdb: detected capacity change from 0 to 1073741824 [ 3.284703] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.287879] vdc: detected capacity change from 0 to 2621440000 [ 3.313775] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.316815] vdd: detected capacity change from 0 to 2621440000 [ 3.336800] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.339295] vde: detected capacity change from 0 to 4294967296 [ 3.359688] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.364488] vdf: detected capacity change from 0 to 4294967296 [ 3.370476] libphy: Fixed MDIO Bus: probed [ 3.376397] usbcore: registered new interface driver usbserial_generic [ 3.379108] usbserial: USB Serial support registered for generic [ 3.382113] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.387607] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.389499] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.392463] mousedev: PS/2 mouse device common for all mice [ 3.395756] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.397469] rtc_cmos 00:05: RTC can wake from S4 [ 3.402994] rtc_cmos 00:05: registered as rtc0 [ 3.403318] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.405077] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.410901] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.411719] intel_pstate: CPU model not supported [ 3.418073] hid: raw HID events driver (C) Jiri Kosina [ 3.420348] usbcore: registered new interface driver usbhid [ 3.422516] usbhid: USB HID core driver [ 3.424213] drop_monitor: Initializing network drop monitor service [ 3.426814] Initializing XFRM netlink socket [ 3.428888] NET: Registered protocol family 10 [ 3.432828] Segment Routing with IPv6 [ 3.434506] NET: Registered protocol family 17 [ 3.436871] mpls_gso: MPLS GSO support [ 3.442301] RAS: Correctable Errors collector initialized. [ 3.444123] AVX version of gcm_enc/dec engaged. [ 3.445960] AES CTR mode by8 optimization enabled [ 3.519508] sched_clock: Marking stable (3519486430, 0)->(4465553039, -946066609) [ 3.522170] registered taskstats version 1 [ 3.523831] Loading compiled-in X.509 certificates [ 3.525951] zswap: loaded using pool lzo/zbud [ 3.548606] Key type big_key registered [ 3.559595] Key type encrypted registered [ 3.561270] ima: No TPM chip found, activating TPM-bypass! [ 3.563078] ima: Allocated hash algorithm: sha1 [ 3.564778] ima: No architecture policies found [ 3.566320] evm: Initialising EVM extended attributes: [ 3.567833] evm: security.selinux [ 3.569028] evm: security.ima [ 3.570197] evm: security.capability [ 3.571521] evm: HMAC attrs: 0x1 [ 3.573531] rtc_cmos 00:05: setting system clock to 2026-08-27 19:25:47 UTC (1787858747) [ 3.579130] debug: unmapping init [mem 0xffffffff92c03000-0xffffffff92dfffff] [ 3.582233] debug: unmapping init [mem 0xffffffff91982000-0xffffffff91c58fff] [ 3.591117] Write protecting the kernel read-only data: 28672k [ 3.594503] debug: unmapping init [mem 0xffffffff90003000-0xffffffff901fffff] [ 3.597790] debug: unmapping init [mem 0xffffffff90914000-0xffffffff909fffff] [ 3.634174] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.642157] systemd[1]: Detected virtualization kvm. [ 3.644148] systemd[1]: Detected architecture x86-64. [ 3.646037] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.668777] systemd[1]: No hostname configured. [ 3.670658] systemd[1]: Set hostname to . [ 3.672902] random: systemd: uninitialized urandom read (16 bytes read) [ 3.675349] systemd[1]: Initializing machine ID from random generator. [ 3.718721] random: ln: uninitialized urandom read (6 bytes read) [ 3.803535] random: systemd: uninitialized urandom read (16 bytes read) [ 3.806785] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.816282] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ 3.824139] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.370777] device-mapper: uevent: version 1.0.3 [ 4.372836] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.046250] virtio_net virtio0 ens2: renamed from eth0 [ 5.059605] random: fast init done [ 5.123360] scsi host0: ata_piix [ 5.168098] scsi host1: ata_piix [ 5.169796] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.172234] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.483352] dracut-initqueue[583]: RTNETLINK answers: File exists [ 10.088207] random: crng init done [ 10.089318] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.402158] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.505387] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.755342] SELinux: Disabled at runtime. [ 11.817394] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.825475] systemd[1]: Detected virtualization kvm. [ 11.827139] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.333395] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.337286] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.341815] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.347290] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.350634] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.358896] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.372868] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. [ 12.459058] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Reached target Paths. Starting udev Coldplug all Devices... Mounting Huge Pages File System... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-getty.slice. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ 12.761760] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /mnt. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.026862] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.037609] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.154217] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.163719] EDAC sbridge: Ver: 1.1.2 [ 14.846652] Key type dns_resolver registered [ 15.152759] NFS: Registering the id_resolver key type [ 15.154918] Key type id_resolver registered [ 15.156641] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg223-server login: [ 52.396027] hrtimer: interrupt took 2484119 ns [ 85.840342] libcfs: loading out-of-tree module taints kernel. [ 85.969509] Key type ._llcrypt registered [ 85.975330] Key type .llcrypt registered [ 86.238604] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 110.662164] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 112.386870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 112.400975] alg: No test for adler32 (adler32-zlib) [ 114.282526] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 115.082871] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 116.848192] Key type lgssc registered [ 119.088317] Lustre: Echo OBD driver; http://www.lustre.org/ [ 142.405640] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 192.838353] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 210.285645] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 210.336491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 211.875915] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 211.932925] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 212.036889] Lustre: lustre-MDT0000: new disk, initializing [ 212.146501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 212.181353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 218.323865] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 234.593579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 234.707310] Lustre: 6508:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 234.766611] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 234.775584] Lustre: Skipped 1 previous similar message [ 234.884648] Lustre: lustre-MDT0001: new disk, initializing [ 234.991658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 235.024945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 235.038645] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 240.460596] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 245.231551] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 257.601284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 258.056311] Lustre: lustre-OST0000: new disk, initializing [ 258.059691] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 258.064328] Lustre: 8448:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 258.171640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 266.130940] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 266.288873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 266.304584] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 266.528820] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 284.759431] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 285.077786] Lustre: lustre-OST0001: new disk, initializing [ 285.093934] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 285.116170] Lustre: 9520:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 285.317621] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 292.612564] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 294.982737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 294.993964] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 295.078632] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 306.523711] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 313.748290] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 321.227480] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing check_logdir /tmp/testlogs/ [ 328.767172] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing yml_node [ 333.458500] Lustre: DEBUG MARKER: Client: 2.17.57.89 [ 336.645484] Lustre: DEBUG MARKER: MDS: 2.17.57.89 [ 339.523487] Lustre: DEBUG MARKER: OSS: 2.17.57.89 [ 341.833897] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 15:31:23 EDT 2026 [ 369.392945] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 371.245358] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 388.579654] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 388.587167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 388.605236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 390.112698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 390.118210] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 390.160115] Lustre: Skipped 2 previous similar messages [ 395.238954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 395.255195] Lustre: Skipped 6 previous similar messages [ 400.354627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 400.371053] Lustre: Skipped 3 previous similar messages [ 401.378386] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 401.950462] Lustre: server umount lustre-MDT0000 complete [ 407.423940] LustreError: 7458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859151 with bad export cookie 6726674388217609132 [ 407.429360] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 407.449500] LustreError: 7458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 410.596154] LustreError: 10097:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 410.607594] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 410.624785] LustreError: 10097:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 6 previous similar messages [ 410.648708] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 410.658051] Lustre: Skipped 1 previous similar message [ 412.641312] LustreError: 6518:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 412.656261] LustreError: 6518:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 414.081452] Lustre: server umount lustre-MDT0001 complete [ 418.568052] Lustre: server umount lustre-OST0000 complete [ 422.766978] Lustre: server umount lustre-OST0001 complete [ 430.971332] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 442.665473] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 491.270994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 491.543835] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 491.573215] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 491.712538] Lustre: lustre-MDT0000: new disk, initializing [ 491.870115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 491.908655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 498.441139] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 510.950171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 511.095251] Lustre: 15082:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 511.184527] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 511.191060] Lustre: Skipped 1 previous similar message [ 511.283968] Lustre: lustre-MDT0001: new disk, initializing [ 511.419929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 511.534814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 511.551972] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 516.300794] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 526.958590] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 532.706472] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 539.896370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 540.138484] Lustre: lustre-OST0000: new disk, initializing [ 540.140113] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 540.143772] Lustre: 17051:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 540.192071] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 541.460410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 541.524098] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 541.605715] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 547.759716] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 559.865809] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 566.376219] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 566.586458] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 572.789723] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 573.000586] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 577.516663] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 577.517768] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 577.537342] Lustre: Skipped 1 previous similar message [ 577.548380] LustreError: Skipped 1 previous similar message [ 577.549239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 577.568963] Lustre: Skipped 3 previous similar messages [ 581.891954] Lustre: server umount lustre-OST0000 complete [ 587.745923] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 587.779547] Lustre: Skipped 2 previous similar messages [ 593.146773] Lustre: server umount lustre-MDT0000 complete [ 597.790614] LustreError: 15075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859341 with bad export cookie 6726674388217610665 [ 597.814431] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 598.365186] Lustre: server umount lustre-MDT0001 complete [ 604.627962] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 15:35:46 (1787859346) [ 627.957992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 628.595601] LustreError: 19929:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 628.696691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 633.826887] LustreError: 19930:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 634.919563] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 638.951127] LustreError: 19929:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 647.533652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 654.133912] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 663.984916] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 671.136626] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 673.603165] Lustre: Failing over lustre-MDT0000 [ 673.614485] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 673.617994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 673.623330] Lustre: Skipped 1 previous similar message [ 673.633703] Lustre: Skipped 5 previous similar messages [ 673.847936] Lustre: server umount lustre-MDT0000 complete [ 678.882988] LustreError: 19930:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 678.900083] LustreError: 19930:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 694.250915] Lustre: 3639:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787859422/real 1787859422] req@ffff9aa1f9a63840 x1874705891110400/t0(0) o400->MGC192.168.202.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787859438 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 694.283380] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 695.142133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 699.362312] LustreError: 19930:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 699.395120] LustreError: 19930:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 9 previous similar messages [ 704.507770] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x5d59f2b7756dc72f [ 704.516534] Lustre: MGC192.168.202.123@tcp: Connection restored to 0@lo (at 0@lo) [ 704.830154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 704.836457] Lustre: Skipped 1 previous similar message [ 704.903527] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 710.115756] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 710.125457] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 710.167270] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 710.496711] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 721.514394] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 727.523868] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 727.554456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 727.572333] Lustre: Skipped 1 previous similar message [ 733.326969] Lustre: server umount lustre-MDT0000 complete [ 737.761626] LustreError: 19929:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 737.804422] LustreError: 19929:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 3 previous similar messages [ 740.004393] LustreError: 19910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859483 with bad export cookie 6726674388217612079 [ 740.009698] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 740.021153] LustreError: 19910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 740.550804] Lustre: server umount lustre-MDT0001 complete [ 751.042382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 758.126763] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 769.024559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 775.363879] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 785.814402] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 793.849522] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 796.615081] Lustre: Failing over lustre-MDT0000 [ 796.929737] Lustre: server umount lustre-MDT0000 complete [ 800.230096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 800.241330] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 800.253311] Lustre: Skipped 1 previous similar message [ 806.378586] LustreError: 23275:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 806.398260] LustreError: 23275:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 8 previous similar messages [ 817.632640] Lustre: 3639:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787859545/real 1787859545] req@ffff9aa1ff9752c0 x1874705891143040/t0(0) o400->MGC192.168.202.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787859561 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 817.662116] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 818.767630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 828.464337] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 833.525830] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 833.543156] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 833.549986] Lustre: Skipped 1 previous similar message [ 833.594390] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 833.711775] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 846.268794] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 853.986804] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 854.004146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 854.007016] Lustre: Skipped 2 previous similar messages [ 857.871669] Lustre: server umount lustre-MDT0000 complete [ 862.673962] LustreError: 23255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859606 with bad export cookie 6726674388217613017 [ 862.681545] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 862.693910] LustreError: 23255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 863.300624] Lustre: server umount lustre-MDT0001 complete [ 874.335211] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 15:40:15 (1787859615) [ 895.387378] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 907.508295] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 954.837894] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 970.600948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 970.995453] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 971.053990] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 971.147583] Lustre: lustre-MDT0000: new disk, initializing [ 971.213332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 971.216168] Lustre: Skipped 3 previous similar messages [ 971.230706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 977.042351] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 990.459487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 990.630500] Lustre: 29657:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 990.967956] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 996.710986] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 1008.313650] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1016.297347] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1025.781419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1026.351885] Lustre: 31630:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1027.830416] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1027.887960] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 1028.054103] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1028.100268] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 1035.268456] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 1047.538237] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1056.533501] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 1084.386036] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1084.401216] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1084.418416] Lustre: Skipped 3 previous similar messages [ 1084.429260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1084.436495] Lustre: Skipped 3 previous similar messages [ 1087.025710] Lustre: server umount lustre-OST0000 complete [ 1093.601839] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1093.622804] LustreError: Skipped 1 previous similar message [ 1099.151673] Lustre: server umount lustre-MDT0000 complete [ 1099.749910] LustreError: 30746:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1099.762824] LustreError: 30746:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 13 previous similar messages [ 1105.594682] LustreError: 30454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859849 with bad export cookie 6726674388217613423 [ 1105.616540] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1105.623072] LustreError: 30454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1106.308709] Lustre: server umount lustre-MDT0001 complete [ 1124.228095] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 1128.794800] Key type lgssc unregistered [ 1129.261488] LNet: 33614:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1129.269728] LNetError: 33614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1129.287095] LNet: Removed LNI 192.168.202.123@tcp [ 1130.826722] Key type .llcrypt unregistered [ 1130.827767] Key type ._llcrypt unregistered [ 1162.437699] Key type ._llcrypt registered [ 1162.442587] Key type .llcrypt registered [ 1162.678677] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 1164.371775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1164.398272] alg: No test for adler32 (adler32-zlib) [ 1165.691880] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 1166.086106] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 1167.832259] Key type lgssc registered [ 1169.575816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1179.255425] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1179.276734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1180.809628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1186.981145] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 1190.376482] LustreError: 35030:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1190.404760] LustreError: 35030:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 1195.495376] LustreError: 35031:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1198.916358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1199.511412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1205.720449] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 1214.994983] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1223.435553] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1232.142169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1232.742025] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1241.847474] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 1245.077242] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1250.269665] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1253.150429] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1272.304566] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 15:46:53 (1787860013) [ 1275.200573] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1278.000928] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1288.018767] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 15:47:09 (1787860029) [ 1299.671698] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 15:47:21 (1787860041) [ 1300.894182] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1311.879692] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1311.887263] Lustre: Skipped 2 previous similar messages [ 1314.065948] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 15:47:35 (1787860055) [ 1316.342838] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1316.351431] Lustre: Skipped 2 previous similar messages [ 1324.683949] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 1324.690793] Lustre: Skipped 1 previous similar message [ 1341.269376] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1341.276656] Lustre: Skipped 5 previous similar messages [ 1345.148873] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 15:48:06 (1787860086) [ 1348.742941] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1353.149963] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1362.633205] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 15:48:24 (1787860104) [ 1378.007611] Lustre: Setting parameter general.jobid_name=jobid_name=jobid_name=%e.root in log params [ 1378.017244] Lustre: Skipped 7 previous similar messages [ 1379.479883] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 15:48:41 (1787860121) [ 1398.757695] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1398.759267] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1398.772126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1398.777824] LustreError: Skipped 1 previous similar message [ 1398.796260] Lustre: Skipped 1 previous similar message [ 1403.886105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1403.897144] Lustre: Skipped 2 previous similar messages [ 1408.993372] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1408.997940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1409.369803] Lustre: server umount lustre-OST0000 complete [ 1414.625507] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1414.631249] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1414.640399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1414.645259] Lustre: Skipped 1 previous similar message [ 1417.199919] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1420.884468] Lustre: server umount lustre-MDT0000 complete [ 1422.317764] LustreError: 35027:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1422.338708] LustreError: 35027:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 1425.407248] LustreError: 37952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860169 with bad export cookie 16829083376811136660 [ 1425.411523] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1425.427648] LustreError: 37952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1426.199651] Lustre: server umount lustre-MDT0001 complete [ 1447.293749] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 1451.356319] Key type lgssc unregistered [ 1451.694617] LNet: 42921:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1451.703632] LNetError: 42921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1451.728794] LNet: Removed LNI 192.168.202.123@tcp [ 1452.565778] Key type .llcrypt unregistered [ 1452.567151] Key type ._llcrypt unregistered [ 1475.018886] Key type ._llcrypt registered [ 1475.022856] Key type .llcrypt registered [ 1475.212679] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 1495.687824] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 1497.555957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1497.609266] alg: No test for adler32 (adler32-zlib) [ 1498.836523] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 1499.094219] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 1500.752159] Key type lgssc registered [ 1502.039320] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1544.823263] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1544.892748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1546.228941] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1546.283265] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1546.447976] Lustre: lustre-MDT0000: new disk, initializing [ 1546.590121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1546.614367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1552.846568] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 1564.391482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1564.537127] Lustre: 46073:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1564.610208] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1564.613886] Lustre: Skipped 1 previous similar message [ 1564.715431] Lustre: lustre-MDT0001: new disk, initializing [ 1564.827582] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1564.875045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1564.905641] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1570.696278] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 1580.394916] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1586.289536] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1593.929660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1594.299309] Lustre: lustre-OST0000: new disk, initializing [ 1594.302789] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1594.309077] Lustre: 48040:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1594.445066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1601.768920] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 1604.216558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1604.227989] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1604.429616] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1612.350936] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1618.305804] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1618.607833] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1624.123765] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1624.499069] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1630.178287] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1630.180255] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1630.189032] LustreError: Skipped 1 previous similar message [ 1630.190803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1630.220335] Lustre: Skipped 1 previous similar message [ 1633.681357] Lustre: server umount lustre-OST0000 complete [ 1640.417632] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1640.422433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1640.430070] Lustre: Skipped 1 previous similar message [ 1640.446375] Lustre: Skipped 2 previous similar messages [ 1645.538394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1645.549198] Lustre: Skipped 1 previous similar message [ 1650.658636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1650.676623] Lustre: Skipped 1 previous similar message [ 1652.192191] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1652.416291] Lustre: server umount lustre-MDT0000 complete [ 1656.681214] LustreError: 46068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860400 with bad export cookie 7053879503558419422 [ 1656.695699] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1657.164535] Lustre: server umount lustre-MDT0001 complete [ 1668.928778] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 15:53:30 (1787860410) [ 1677.175097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1677.624341] LustreError: 50506:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1677.631791] LustreError: 50506:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 3 previous similar messages [ 1677.705315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1682.917068] LustreError: 50507:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1683.248865] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 1688.051330] LustreError: 50506:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1693.169264] LustreError: 50507:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1693.968163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1694.391524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1700.252661] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 1711.062996] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1717.411869] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1725.735209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1726.299551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1733.869244] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 1742.977530] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1752.958627] LustreError: 53039:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1753.822580] Lustre: Modifying parameter general.timeout=1 in log params [ 1755.452987] Lustre: Modifying parameter general.timeout=3 in log params [ 1755.462314] Lustre: Skipped 1 previous similar message [ 1757.602970] Lustre: Modifying parameter general.timeout=5 in log params [ 1757.606661] Lustre: Skipped 1 previous similar message [ 1761.683267] Lustre: Modifying parameter general.timeout=9 in log params [ 1761.685488] Lustre: Skipped 3 previous similar messages [ 1769.924364] Lustre: Modifying parameter general.timeout=17 in log params [ 1769.932211] Lustre: Skipped 7 previous similar messages [ 1787.002994] Lustre: Modifying parameter general.timeout=33 in log params [ 1787.011616] Lustre: Skipped 15 previous similar messages [ 1819.110188] Lustre: Modifying parameter general.timeout=65 in log params [ 1819.112740] Lustre: Skipped 31 previous similar messages [ 1883.517610] Lustre: Modifying parameter general.timeout=125 in log params [ 1883.530748] Lustre: Skipped 59 previous similar messages [ 1900.661821] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 15:57:22 (1787860642) [ 1903.874059] Lustre: 59994:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1905.511831] Lustre: 60093:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1907.534437] Lustre: 60191:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1911.548204] Lustre: 60401:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1911.557712] Lustre: 60401:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1947.479515] Lustre: 61860:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1947.483110] Lustre: 61860:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1947.497809] LustreError: 61860:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1951.536378] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 15:58:13 (1787860693) [ 1969.637904] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1969.647668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1969.656226] Lustre: Skipped 2 previous similar messages [ 1969.668172] Lustre: Skipped 2 previous similar messages [ 1972.042377] Lustre: server umount lustre-MDT0000 complete [ 1974.754091] LustreError: 51940:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1974.787073] LustreError: 51940:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 1976.620137] LustreError: 50488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860720 with bad export cookie 7053879503558420297 [ 1976.632987] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1976.633973] LustreError: 50488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 1977.107626] Lustre: server umount lustre-MDT0001 complete [ 1991.868865] Lustre: server umount lustre-OST0000 complete [ 2022.554199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2022.959643] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2022.986663] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2022.992903] Lustre: Skipped 27 previous similar messages [ 2023.411145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2029.532544] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2039.794170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2039.926706] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2039.963853] Lustre: 64393:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2044.685325] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2055.603544] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2063.913284] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2071.882090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2071.995628] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2072.331210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2072.336798] Lustre: Skipped 1 previous similar message [ 2080.267705] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2090.547912] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2093.411082] Lustre: 66964:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 2105.316841] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2105.328639] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2105.337534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2107.287438] Lustre: server umount lustre-OST0000 complete [ 2111.976710] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2111.981383] LustreError: Skipped 1 previous similar message [ 2111.984132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2111.994419] Lustre: Skipped 1 previous similar message [ 2117.856908] Lustre: server umount lustre-MDT0000 complete [ 2121.186028] LustreError: 64400:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2121.208849] LustreError: 64400:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 2122.437959] LustreError: 64386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860866 with bad export cookie 7053879503558423454 [ 2122.450166] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2122.451249] LustreError: 64386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2122.862037] Lustre: server umount lustre-MDT0001 complete [ 2135.641774] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 2139.398454] Key type lgssc unregistered [ 2139.889509] LNet: 68191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2139.907417] LNetError: 68191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2139.937659] LNet: Removed LNI 192.168.202.123@tcp [ 2140.808494] Key type .llcrypt unregistered [ 2140.811604] Key type ._llcrypt unregistered [ 2169.921830] Key type ._llcrypt registered [ 2169.929210] Key type .llcrypt registered [ 2170.077926] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 2172.134255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2172.260090] alg: No test for adler32 (adler32-zlib) [ 2173.425559] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 2173.767954] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 2175.488178] Key type lgssc registered [ 2177.262753] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2188.897531] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2188.909204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2190.532702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2196.283251] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2201.075820] LustreError: 69622:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2201.094987] LustreError: 69622:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 2206.178771] LustreError: 69623:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2206.855450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2207.577368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2212.924390] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2222.908557] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2231.185407] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2239.443000] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2240.409538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2247.996121] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2256.964539] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2269.778271] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2282.474376] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2282.482973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2282.505473] Lustre: Skipped 2 previous similar messages [ 2282.542693] Lustre: Skipped 2 previous similar messages [ 2287.587223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2287.601103] Lustre: Skipped 2 previous similar messages [ 2288.612403] Lustre: server umount lustre-MDT0000 complete [ 2292.712274] LustreError: 69623:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2292.741708] LustreError: 69623:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 2293.679323] LustreError: 72308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861037 with bad export cookie 1542364032863681800 [ 2293.698202] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2293.699873] LustreError: 72308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2294.363293] Lustre: server umount lustre-MDT0001 complete [ 2310.774455] Lustre: server umount lustre-OST0000 complete [ 2346.025171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2346.463908] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2346.509863] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2346.997119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2351.481424] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2365.613050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2365.860435] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2365.937086] Lustre: 74498:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2366.859109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2376.052949] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2387.344812] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2394.973408] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2403.578688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2403.894608] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2404.729421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2413.482224] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2422.791778] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2424.410830] Lustre: 77076:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2426.538798] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2434.540210] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2434.543935] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2434.576246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2438.945736] Lustre: server umount lustre-OST0000 complete [ 2444.771439] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2444.782668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2444.792364] Lustre: Skipped 2 previous similar messages [ 2444.808717] Lustre: Skipped 2 previous similar messages [ 2449.889280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2449.898419] Lustre: Skipped 1 previous similar message [ 2450.029979] Lustre: server umount lustre-MDT0000 complete [ 2454.232361] LustreError: 74488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861198 with bad export cookie 1542364032863682647 [ 2454.243636] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2454.248337] LustreError: 74488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2454.642553] Lustre: server umount lustre-MDT0001 complete [ 2467.189613] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 2470.904930] Key type lgssc unregistered [ 2471.267729] LNet: 78291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2471.273670] LNetError: 78291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2471.287626] LNet: Removed LNI 192.168.202.123@tcp [ 2472.335833] Key type .llcrypt unregistered [ 2472.337784] Key type ._llcrypt unregistered [ 2500.926045] Key type ._llcrypt registered [ 2500.930780] Key type .llcrypt registered [ 2501.035193] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 2502.602092] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2502.617930] alg: No test for adler32 (adler32-zlib) [ 2503.765518] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 2504.029954] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 2505.752582] Key type lgssc registered [ 2507.137445] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2515.991378] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2516.016072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2517.763843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2522.715525] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2527.207316] LustreError: 79706:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2527.224920] LustreError: 79706:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 2532.333842] LustreError: 79707:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2533.511675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2534.505434] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2540.441558] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2549.647780] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2555.769277] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2563.607091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2564.417367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2571.957780] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2581.520467] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2599.060165] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2600.208383] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2601.247793] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2603.613657] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2603.620039] Lustre: Skipped 1 previous similar message [ 2607.994380] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2607.998566] Lustre: Skipped 3 previous similar messages [ 2616.197687] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2616.202127] Lustre: Skipped 7 previous similar messages [ 2632.821805] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2632.826065] Lustre: Skipped 16 previous similar messages [ 2665.463357] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2665.467321] Lustre: Skipped 30 previous similar messages [ 2712.320284] Lustre: 87843:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2715.611480] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 16:10:57 (1787861457) [ 2717.654206] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2719.694716] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 16:11:01 (1787861461) [ 2725.349991] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2725.371810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2725.381928] Lustre: Skipped 1 previous similar message [ 2730.469576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2730.473817] Lustre: Skipped 1 previous similar message [ 2731.201191] Lustre: server umount lustre-OST0000 complete [ 2735.589349] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2735.592752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2735.595294] Lustre: Skipped 2 previous similar messages [ 2735.611533] Lustre: Skipped 1 previous similar message [ 2740.544908] Lustre: server umount lustre-MDT0000 complete [ 2740.730957] LustreError: 79703:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2740.779228] LustreError: 79703:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 2744.738983] LustreError: 81559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861488 with bad export cookie 7197185733247909750 [ 2744.747555] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2745.268979] Lustre: server umount lustre-MDT0001 complete [ 2762.212408] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 2765.255977] Key type lgssc unregistered [ 2765.577783] LNet: 89101:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2765.584762] LNetError: 89101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2765.628315] LNet: Removed LNI 192.168.202.123@tcp [ 2766.607903] Key type .llcrypt unregistered [ 2766.611524] Key type ._llcrypt unregistered [ 2774.414486] Key type ._llcrypt registered [ 2774.415943] Key type .llcrypt registered [ 2774.667937] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2779.882760] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules [ 2782.559510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2782.628778] alg: No test for adler32 (adler32-zlib) [ 2783.632370] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 2783.797295] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 2785.464208] Key type lgssc registered [ 2787.600870] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2789.551688] LustreError: 90327:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2810.650163] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 2811.293640] LustreError: 90327:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2822.716695] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2822.793332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2824.390555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2830.043764] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2834.918282] LustreError: 91328:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2834.936654] LustreError: 91328:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 2840.039494] LustreError: 91328:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2841.329669] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 16:13:02 (1787861582) [ 2845.153933] LustreError: 91329:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2847.703790] LustreError: 91360:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2847.716792] LustreError: 91360:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2847.729338] LustreError: 91360:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 23, retries 0, failed: rc = -5 [ 2848.114315] Lustre: server umount lustre-MDT0000 complete [ 2864.433391] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 2868.036360] Key type lgssc unregistered [ 2868.376912] LNet: 92859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2868.400361] LNetError: 92859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2868.416687] LNet: Removed LNI 192.168.202.123@tcp [ 2869.339587] Key type .llcrypt unregistered [ 2869.342660] Key type ._llcrypt unregistered [ 2889.220504] Key type ._llcrypt registered [ 2889.221899] Key type .llcrypt registered [ 2889.396846] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 2890.514780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2890.610486] alg: No test for adler32 (adler32-zlib) [ 2891.645325] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 2892.002302] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 2893.738325] Key type lgssc registered [ 2894.968633] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2903.684682] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2903.708455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2905.380646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2910.855985] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2915.825327] LustreError: 94078:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2915.842536] LustreError: 94078:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 2920.402669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2921.415546] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2926.638499] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2935.253579] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2941.077208] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2950.937143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2952.312103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2960.640648] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2971.107266] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3026.296402] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 16:16:08 (1787861768) [ 3027.983448] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 3029.806137] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 16:16:11 (1787861771) [ 3037.669216] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3037.681178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3037.692440] Lustre: Skipped 2 previous similar messages [ 3037.716877] Lustre: Skipped 2 previous similar messages [ 3040.434442] Lustre: server umount lustre-MDT0000 complete [ 3042.796044] LustreError: 94079:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3042.828952] LustreError: 94079:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 3044.884430] LustreError: 94060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861788 with bad export cookie 5138047393333503118 [ 3044.895571] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3044.904187] LustreError: 94060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3045.145720] Lustre: server umount lustre-MDT0001 complete [ 3059.454648] Lustre: server umount lustre-OST0000 complete [ 3074.232859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3074.792532] LustreError: 98375:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3074.936860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3080.202499] LustreError: 98376:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3080.321742] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3084.327565] LustreError: 98375:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3092.904849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3093.762094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3100.002322] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3110.951901] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3117.827451] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3133.550293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3133.805651] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 3133.812405] LustreError: 98364:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 3133.838656] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 3133.848816] LustreError: 100461:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 3133.857527] LustreError: 100461:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 3133.865376] LustreError: 100461:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 3133.915809] Lustre: server umount lustre-OST0000 complete [ 3133.918658] LustreError: 100461:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3143.657692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3143.949499] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 3143.957705] LustreError: 98363:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 3144.023105] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 3144.037949] LustreError: 100927:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 3144.042670] LustreError: 100927:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 3144.047583] LustreError: 100927:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 3144.143299] Lustre: server umount lustre-OST0000 complete [ 3144.156378] LustreError: 100927:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3155.204444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3155.534640] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3155.542673] Lustre: Found index 0 for lustre-OST0000, updating log [ 3155.553158] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 3155.723605] Lustre: lustre-OST0000: new disk, initializing [ 3155.732545] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3155.740264] Lustre: 101462:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3156.057439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3163.545299] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3167.779124] Lustre: Failing over lustre-OST0000 [ 3167.969159] Lustre: server umount lustre-OST0000 complete [ 3175.396455] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3175.401496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3175.409355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3181.025547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3181.036434] Lustre: Skipped 2 previous similar messages [ 3184.096170] Lustre: 93391:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787861912/real 1787861912] req@00000000dd4d1ac4 x1874708801888128/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787861928 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3184.130073] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3184.140349] Lustre: Skipped 1 previous similar message [ 3184.147695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3184.158148] Lustre: Skipped 1 previous similar message [ 3185.056139] Lustre: 93388:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787861912/real 1787861912] req@000000007e6cfffc x1874708801887872/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787861928 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3186.081298] Lustre: 93388:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787861914/real 1787861914] req@000000002c58f5f4 x1874708801888896/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787861930 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3186.144151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3186.241041] Lustre: server umount lustre-MDT0000 complete [ 3189.221188] LustreError: 98375:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3189.246666] LustreError: 98375:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 3190.226407] LustreError: 98355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861934 with bad export cookie 5138047393333505862 [ 3190.229046] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3190.236219] LustreError: 98355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3190.462924] Lustre: server umount lustre-MDT0001 complete [ 3202.841277] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 16:19:03 (1787861943) [ 3227.626847] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 3242.295556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3242.665147] LustreError: 103830:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3242.746562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3248.932847] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3259.169028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3265.827409] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3269.325334] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3276.914323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3277.690599] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3277.696803] Lustre: Skipped 1 previous similar message [ 3277.719460] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3284.843181] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3284.909229] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3284.910931] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3286.278493] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3296.548852] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3296.652766] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3296.742512] Lustre: lustre-OST0001: new disk, initializing [ 3296.745850] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3296.751169] Lustre: 106149:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3304.395737] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3306.163929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3306.179243] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3306.312559] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3312.849723] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3321.267859] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3326.957926] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3326.966771] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3326.974835] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3326.988387] Lustre: Skipped 1 previous similar message [ 3337.193046] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3337.195772] Lustre: Skipped 7 previous similar messages [ 3341.280196] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3341.491810] Lustre: server umount lustre-MDT0001 complete [ 3342.355468] LustreError: 103831:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3342.381388] LustreError: 103831:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 5 previous similar messages [ 3353.950553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3354.124692] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3354.131360] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3354.186630] Lustre: 103818:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3355.376188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3355.385278] Lustre: Skipped 1 previous similar message [ 3356.041128] LustreError: 107690:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3356.055265] LustreError: 107690:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3356.072446] LustreError: 103822:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3356.204488] LustreError: 107695:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.123@tcp: cfg command failed: rc = -17 [ 3356.216269] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3356.216269] [ 3360.776692] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3362.797138] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3376.939399] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 16:21:58 (1787862118) [ 3402.949858] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 3426.673133] Lustre: 109790:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3455.444558] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3460.878520] Lustre: 110926:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3564.513825] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3564.530499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3564.548478] Lustre: Skipped 1 previous similar message [ 3564.552623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3564.557197] Lustre: Skipped 1 previous similar message [ 3565.556655] Lustre: server umount lustre-MDT0000 complete [ 3565.561905] LustreError: 106941:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3565.561930] LustreError: 106941:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 9 previous similar messages [ 3570.047074] LustreError: 104971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862313 with bad export cookie 5138047393333506863 [ 3570.055249] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3570.057697] LustreError: 104971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3570.437505] Lustre: server umount lustre-MDT0001 complete [ 3585.835344] Lustre: server umount lustre-OST0000 complete [ 3587.042459] Lustre: 93389:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787862314/real 1787862314] req@ffff9aa1dd150b40 x1874708802181632/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787862330 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3587.093623] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3587.112856] Lustre: Skipped 3 previous similar messages [ 3590.148038] Lustre: server umount lustre-OST0001 complete [ 3597.311038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3609.632732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3619.938828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3633.144740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3658.898580] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 3670.114719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3670.130508] Lustre: lustre-MDT0000: reset Object Index mappings [ 3670.608462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3675.469473] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3685.772691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3685.795209] Lustre: lustre-MDT0001: reset Object Index mappings [ 3692.447905] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3696.223526] Lustre: 116027:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3704.374087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3711.281198] LustreError: 116385:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3711.299382] LustreError: 116385:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 7 previous similar messages [ 3711.309991] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:232 to 0x280000400:257) [ 3711.436054] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3716.406905] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 3720.938323] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3727.369446] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3727.382080] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3730.784447] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3741.376756] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3747.013079] Lustre: 117904:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3805.931913] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 16:29:07 (1787862547) [ 3834.851378] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3834.876227] Lustre: Skipped 1 previous similar message [ 3834.876594] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3834.877822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3838.597891] Lustre: server umount lustre-MDT0000 complete [ 3843.272110] LustreError: 114862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862587 with bad export cookie 5138047393333553812 [ 3843.275945] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3843.286553] LustreError: 114862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3849.821492] Lustre: server umount lustre-MDT0001 complete [ 3860.771397] Lustre: server umount lustre-OST0000 complete [ 3879.392170] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3879.645814] Lustre: server umount lustre-OST0001 complete [ 3888.290887] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 3901.350173] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 3948.686785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3949.130625] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3949.187235] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3949.326563] Lustre: lustre-MDT0000: new disk, initializing [ 3949.447194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3949.457621] Lustre: Skipped 3 previous similar messages [ 3949.501822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3955.406798] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3965.167776] Lustre: server umount lustre-MDT0000 complete [ 3977.252843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3977.706693] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3977.745525] Lustre: 122182:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3977.766604] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3983.030258] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3989.298366] Lustre: server umount lustre-MDT0000 complete [ 4001.848268] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 16:32:23 (1787862743) [ 4003.960992] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 4007.382749] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 16:32:28 (1787862748) [ 4030.863875] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 4046.752071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4053.642767] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 4064.052137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4064.284925] Lustre: 124237:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4064.334687] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4064.337476] Lustre: Skipped 1 previous similar message [ 4064.381539] Lustre: lustre-MDT0001: new disk, initializing [ 4064.501598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4064.507636] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4069.045136] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 4074.837903] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4082.232043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4082.616444] Lustre: lustre-OST0000: new disk, initializing [ 4082.619944] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4082.625747] Lustre: 125815:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4083.785958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4083.798543] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4083.968680] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4090.236553] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 4101.441630] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4101.591272] Lustre: lustre-OST0001: new disk, initializing [ 4101.594828] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4101.606919] Lustre: 126685:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4103.261171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4103.300796] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4103.508132] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4110.148237] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 4120.960731] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4124.630695] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4578.453095] Lustre: 133110:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4578.461129] Lustre: 133110:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4578.468618] Lustre: 133110:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4578.472618] Lustre: 133110:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4578.476606] Lustre: 133110:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4578.480994] Lustre: 133110:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4888.500151] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 16:47:09 (1787863629) [ 4904.418625] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4904.435553] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4904.447077] Lustre: Skipped 4 previous similar messages [ 4904.456933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4904.467848] Lustre: Skipped 7 previous similar messages [ 4909.595358] Lustre: server umount lustre-MDT0000 complete [ 4913.126177] LustreError: 127518:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4913.149944] LustreError: 127518:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 13 previous similar messages [ 4913.640394] LustreError: 133135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863657 with bad export cookie 5138047393333622692 [ 4913.651157] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4913.652826] LustreError: 133135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4918.242104] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4918.257519] Lustre: Skipped 4 previous similar messages [ 4918.271168] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4918.281462] Lustre: Skipped 4 previous similar messages [ 4920.397578] Lustre: server umount lustre-MDT0001 complete [ 4930.744654] Lustre: server umount lustre-OST0000 complete [ 4941.740153] Lustre: server umount lustre-OST0001 complete [ 4949.810782] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 4958.786697] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 5000.817747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5001.315084] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5001.362521] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5001.586845] Lustre: lustre-MDT0000: new disk, initializing [ 5001.678494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5001.698056] Lustre: Skipped 5 previous similar messages [ 5001.718941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5010.001770] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 5025.375243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5025.596914] Lustre: 136171:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5025.681457] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5025.684250] Lustre: Skipped 1 previous similar message [ 5025.765820] Lustre: lustre-MDT0001: new disk, initializing [ 5026.013684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5026.029908] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5031.377559] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 5041.848293] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5048.454799] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5055.819251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5056.232648] Lustre: lustre-OST0000: new disk, initializing [ 5056.236403] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5056.242743] Lustre: 138141:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5057.721209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5057.735956] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5057.890253] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5063.469856] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 5074.068830] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5086.472326] Lustre: lustre-MDD0000: changelog on [ 5090.508530] Lustre: lustre-MDD0001: changelog on [ 6974.516556] Lustre: lustre-MDD0001: changelog off [ 6976.431478] Lustre: 136180:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6977.258796] Lustre: lustre-MDD0000: changelog off [ 6988.261342] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6988.271272] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6988.292831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6988.313368] Lustre: Skipped 1 previous similar message [ 6989.697443] Lustre: server umount lustre-OST0000 complete [ 6996.965096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6996.979307] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6997.001884] Lustre: Skipped 1 previous similar message [ 6997.013042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7003.619046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7003.628340] Lustre: Skipped 3 previous similar messages [ 7008.224297] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7008.723486] Lustre: server umount lustre-MDT0000 complete [ 7008.744139] LustreError: 137591:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7008.762464] LustreError: 137591:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 3 previous similar messages [ 7013.100291] LustreError: 136968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865757 with bad export cookie 5138047393334448167 [ 7013.109678] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7013.120503] LustreError: 136968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7013.631133] Lustre: server umount lustre-MDT0001 complete [ 7029.593633] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 7033.326687] Key type lgssc unregistered [ 7033.688979] LNet: 147293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7033.696844] LNetError: 147293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7033.722218] LNet: Removed LNI 192.168.202.123@tcp [ 7034.768540] Key type .llcrypt unregistered [ 7034.770381] Key type ._llcrypt unregistered [ 7056.793766] Key type ._llcrypt registered [ 7056.797443] Key type .llcrypt registered [ 7056.976071] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 7075.136459] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 7076.780782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7077.098251] alg: No test for adler32 (adler32-zlib) [ 7078.262386] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 7078.491934] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 7080.224186] Key type lgssc registered [ 7082.447680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7120.101650] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7120.138729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7121.447462] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7121.495480] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7121.606462] Lustre: lustre-MDT0000: new disk, initializing [ 7121.816487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7121.833517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7126.784903] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7136.909415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7137.059961] Lustre: 150494:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7137.113472] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7137.118106] Lustre: Skipped 1 previous similar message [ 7137.191358] Lustre: lustre-MDT0001: new disk, initializing [ 7137.279954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7137.344987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7137.361721] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7142.136327] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7152.401675] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7157.762577] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7165.722366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7166.048537] Lustre: lustre-OST0000: new disk, initializing [ 7166.050907] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7166.056620] Lustre: 152465:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7166.139988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7172.402655] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7172.775441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7172.785820] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7172.972380] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7181.756746] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7188.432662] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_module obdecho/obdecho [ 7190.683500] LustreError: 153469:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7190.692498] LustreError: 153469:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7193.094742] LustreError: 153565:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 7208.939486] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7208.941719] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7208.970501] Lustre: Skipped 1 previous similar message [ 7208.977670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7209.008413] Lustre: Skipped 2 previous similar messages [ 7212.186089] Lustre: server umount lustre-MDT0000 complete [ 7217.208061] LustreError: 152464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865961 with bad export cookie 9595810769712987553 [ 7217.217258] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7217.222647] LustreError: 152464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7217.897730] Lustre: server umount lustre-MDT0001 complete [ 7232.605565] Lustre: server umount lustre-OST0000 complete [ 7260.672609] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 7274.343694] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 7314.525818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7314.802994] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7314.850167] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7314.961472] Lustre: lustre-MDT0000: new disk, initializing [ 7315.112140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7315.142456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7320.557516] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7330.370514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7330.430483] Lustre: 157282:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7330.478323] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7330.482282] Lustre: Skipped 1 previous similar message [ 7330.583353] Lustre: lustre-MDT0001: new disk, initializing [ 7330.656409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7330.682469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7330.704672] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7335.171864] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7345.508468] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7351.471314] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7358.895082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7359.213606] Lustre: lustre-OST0000: new disk, initializing [ 7359.215747] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7359.221212] Lustre: 159252:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7359.296924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7364.709309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7364.727847] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7364.860608] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7366.327206] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7378.357832] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7383.504287] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7383.690452] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7388.536202] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7388.801993] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7395.813562] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7395.824757] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7395.837734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7395.871374] Lustre: Skipped 2 previous similar messages [ 7397.292236] Lustre: server umount lustre-OST0000 complete [ 7402.465880] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7402.478554] LustreError: Skipped 1 previous similar message [ 7402.483212] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7402.499715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7402.512240] Lustre: Skipped 1 previous similar message [ 7406.050189] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7406.052562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7406.088131] Lustre: Skipped 1 previous similar message [ 7407.297890] Lustre: server umount lustre-MDT0000 complete [ 7411.171367] LustreError: 157290:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7411.188506] LustreError: 157290:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 5 previous similar messages [ 7411.260925] LustreError: 158077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866155 with bad export cookie 9595810769712988512 [ 7411.268859] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7411.269930] LustreError: 158077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7411.636312] Lustre: server umount lustre-MDT0001 complete [ 7414.860279] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 17:29:17 (1787866157) [ 7416.422466] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7418.146484] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 17:29:20 (1787866160) [ 7426.301565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7426.922584] LustreError: 161664:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7426.979789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7431.714562] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7432.164616] LustreError: 161665:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7436.258342] LustreError: 161664:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7440.367018] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7445.496398] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7453.631489] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7459.952571] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7467.422375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7467.830750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7467.835638] Lustre: Skipped 1 previous similar message [ 7474.064853] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7483.146860] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7495.009027] LustreError: 161691:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x240000bd0:0x2:0x0]: rc = -108 [ 7495.030352] Lustre: 161691:0:(llog_cat_server.c:670:llog_cat_cancel_arr_rec()) lustre-MDT0001-osp-MDT0000: fail to cancel 1 records in [0x240000bd0:0x2:0x0]: rc = -108 [ 7495.663947] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7495.685546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7495.705349] Lustre: Skipped 1 previous similar message [ 7495.738991] Lustre: Skipped 2 previous similar messages [ 7501.907181] Lustre: server umount lustre-MDT0000 complete [ 7504.647396] LustreError: 161659:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7507.975906] LustreError: 161646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866251 with bad export cookie 9595810769712989387 [ 7507.988660] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7507.995331] LustreError: 161646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7509.792658] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.23@tcp (stopping) [ 7509.799200] Lustre: Skipped 4 previous similar messages [ 7511.023967] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7511.040661] Lustre: Skipped 1 previous similar message [ 7514.357163] Lustre: server umount lustre-MDT0001 complete [ 7520.936736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7533.633244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7533.867539] LustreError: 165507:0:(llog.c:1655:llog_backup()) MGC192.168.202.123@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7533.877793] Lustre: 165507:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.123@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7537.699899] LustreError: 165512:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7537.711491] LustreError: 165512:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 6 previous similar messages [ 7555.851633] LustreError: 165514:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7555.882065] LustreError: 165514:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 14 previous similar messages [ 7590.666234] LustreError: 165512:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7590.679249] LustreError: 165512:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 26 previous similar messages [ 7594.465408] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7594.474923] Lustre: 165507:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.123@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7635.455920] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7639.480206] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7640.558862] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7647.680796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7652.833705] LustreError: 148717:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c5c9d387 x1874713192302720/t0(0) o250->MGC192.168.202.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7657.300274] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7658.480311] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7664.583703] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7665.925521] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7671.441777] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7672.992667] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7694.308154] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7694.310811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7694.310941] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7694.327566] Lustre: Skipped 2 previous similar messages [ 7694.358172] Lustre: Skipped 4 previous similar messages [ 7696.104710] Lustre: server umount lustre-MDT0000 complete [ 7699.435767] LustreError: 165513:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7699.460447] LustreError: 165513:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 47 previous similar messages [ 7700.157123] LustreError: 161645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866444 with bad export cookie 9595810769712995848 [ 7700.160446] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7700.165018] LustreError: 161645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7700.451935] Lustre: server umount lustre-MDT0001 complete [ 7714.760319] Lustre: server umount lustre-OST0000 complete [ 7733.889530] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 7742.923194] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 7776.585080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7776.779863] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7776.832541] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7776.914865] Lustre: lustre-MDT0000: new disk, initializing [ 7776.985104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7776.988236] Lustre: Skipped 1 previous similar message [ 7777.006000] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7781.150408] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7790.013547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7790.092268] Lustre: 170613:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7790.264481] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7794.851948] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7802.944424] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7807.696438] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7814.746890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7815.003848] Lustre: lustre-OST0000: new disk, initializing [ 7815.018115] Lustre: Skipped 1 previous similar message [ 7815.026994] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7815.040499] Lustre: Skipped 2 previous similar messages [ 7815.044873] Lustre: 172580:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7820.863424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7820.875713] Lustre: Skipped 1 previous similar message [ 7820.884494] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7821.006402] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7821.220768] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7831.319400] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7835.712417] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7835.893401] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7840.191125] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7840.395803] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7846.887542] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7846.900351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7846.907182] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7848.927690] Lustre: server umount lustre-OST0000 complete [ 7867.360711] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7867.650567] Lustre: server umount lustre-MDT0000 complete [ 7870.993943] LustreError: 170605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866614 with bad export cookie 9595810769713012676 [ 7871.002281] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7871.008277] LustreError: 170605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7871.234784] Lustre: server umount lustre-MDT0001 complete [ 7879.892701] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 17:37:01 (1787866621) [ 7887.327495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7887.654295] LustreError: 174991:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7891.862529] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7899.752695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7904.776165] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7911.281925] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7916.364847] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7922.696860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7930.145738] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7937.527735] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7946.382479] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 17:38:08 (1787866688) [ 7950.818231] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7950.822513] LustreError: Skipped 1 previous similar message [ 7950.829440] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7950.845667] Lustre: Skipped 3 previous similar messages [ 7950.852304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7950.854895] Lustre: Skipped 9 previous similar messages [ 7956.873088] Lustre: server umount lustre-OST0000 complete [ 7968.126996] Lustre: server umount lustre-MDT0000 complete [ 7971.884872] LustreError: 175726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866715 with bad export cookie 9595810769713013565 [ 7971.895972] LustreError: 175726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7971.898700] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7972.406251] Lustre: server umount lustre-MDT0001 complete [ 7984.107422] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 7986.880843] Key type lgssc unregistered [ 7987.119197] LNet: 178722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7987.140135] LNetError: 178722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7987.158526] LNet: Removed LNI 192.168.202.123@tcp [ 7987.843313] Key type .llcrypt unregistered [ 7987.845356] Key type ._llcrypt unregistered [ 8005.483793] Key type ._llcrypt registered [ 8005.494198] Key type .llcrypt registered [ 8005.606280] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 8006.797989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8006.833844] alg: No test for adler32 (adler32-zlib) [ 8007.815765] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 8008.008525] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 8009.648731] Key type lgssc registered [ 8010.651379] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8017.521835] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8017.535425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8033.056428] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8038.176802] LustreError: 179966:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.123@tcp: failed processing log, type 4: rc = -110 [ 8043.361490] LustreError: 179966:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8043.373974] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8043.380768] LustreError: MGC192.168.202.123@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8043.403793] LustreError: 179966:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8043.413577] LustreError: 179966:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 8043.418989] LustreError: 179966:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8043.500689] Lustre: server umount lustre-OST0000 complete [ 8043.504281] LustreError: 179966:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8052.480571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8054.223440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8060.344433] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8064.482729] LustreError: 180466:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8064.499856] LustreError: 180466:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 8068.958698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8069.263416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8073.301621] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8080.495777] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8086.344631] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8094.371366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8094.566186] LustreError: 182283:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8094.583656] Lustre: 182283:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.123@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8094.796952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8100.936822] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8109.733988] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8114.418390] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8114.437909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8114.457044] Lustre: Skipped 2 previous similar messages [ 8114.492939] Lustre: Skipped 2 previous similar messages [ 8119.025674] Lustre: server umount lustre-MDT0000 complete [ 8119.788120] LustreError: 180467:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8119.829858] LustreError: 180467:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 8124.013711] LustreError: 180445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866867 with bad export cookie 9868692293235620725 [ 8124.024876] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8124.728296] LustreError: 181197:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000004ef6756e x1874714166323840/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8124.899757] LustreError: 182956:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8124.927824] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8125.123588] Lustre: server umount lustre-MDT0001 complete [ 8132.012828] Lustre: server umount lustre-OST0000 complete [ 8143.006883] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 8155.320604] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 8196.173039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8196.454257] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8196.494615] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8196.610351] Lustre: lustre-MDT0000: new disk, initializing [ 8196.710870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8196.726522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8201.637351] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8212.222305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8212.350697] Lustre: 185808:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8212.405502] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8212.411065] Lustre: Skipped 1 previous similar message [ 8212.465721] Lustre: lustre-MDT0001: new disk, initializing [ 8212.534714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8212.564047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8212.578381] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8218.310936] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8228.111510] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8233.941568] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8240.727250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8240.947797] Lustre: lustre-OST0000: new disk, initializing [ 8240.951292] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8240.958608] Lustre: 187778:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8241.042099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8247.159380] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8250.986210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8251.001249] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8251.138102] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8257.072580] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8262.237629] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8262.517702] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8267.033693] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8267.265318] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8271.850906] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8271.852087] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8271.862501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8271.867042] LustreError: Skipped 1 previous similar message [ 8271.879994] Lustre: Skipped 1 previous similar message [ 8275.337491] Lustre: server umount lustre-OST0000 complete [ 8279.009420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8279.016796] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8279.045668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8279.054333] Lustre: Skipped 1 previous similar message [ 8282.086217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8282.093696] Lustre: Skipped 1 previous similar message [ 8287.202302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8287.217779] Lustre: Skipped 1 previous similar message [ 8293.344636] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8293.750962] Lustre: server umount lustre-MDT0000 complete [ 8296.421135] LustreError: 185817:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8296.438823] LustreError: 185817:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 8297.362872] LustreError: 186613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867041 with bad export cookie 9868692293235621446 [ 8297.371922] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8297.382836] LustreError: 186613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8297.953675] Lustre: server umount lustre-MDT0001 complete [ 8306.376472] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 17:44:08 (1787867048) [ 8323.034176] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 8325.640250] Key type lgssc unregistered [ 8325.962626] LNet: 190726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8325.968314] LNetError: 190726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8325.990256] LNet: Removed LNI 192.168.202.123@tcp [ 8326.615919] Key type .llcrypt unregistered [ 8326.617664] Key type ._llcrypt unregistered [ 8342.439758] Key type ._llcrypt registered [ 8342.442754] Key type .llcrypt registered [ 8342.577714] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 8343.459743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8343.602074] alg: No test for adler32 (adler32-zlib) [ 8344.695506] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 8344.902586] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 8346.600175] Key type lgssc registered [ 8347.613718] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8356.410870] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8356.481430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8358.172430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8364.182168] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8367.593968] LustreError: 191946:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8367.622152] LustreError: 191946:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 8372.707678] LustreError: 191947:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8373.276265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8373.807891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8379.246987] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8387.942742] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8394.851905] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8403.183768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8403.377463] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8403.383894] LustreError: 193812:0:(llog.c:1696:llog_backup()) MGC192.168.202.123@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8403.391881] Lustre: 193812:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.123@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8403.540235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8410.859619] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8419.973771] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8423.909287] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8423.917976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8423.928709] Lustre: Skipped 1 previous similar message [ 8423.965734] Lustre: Skipped 2 previous similar messages [ 8428.273721] Lustre: server umount lustre-MDT0000 complete [ 8429.026346] LustreError: 192890:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8429.040202] LustreError: 192890:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 8432.444972] LustreError: 191929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867176 with bad export cookie 621702579802975471 [ 8432.459927] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8432.865622] Lustre: server umount lustre-MDT0001 complete [ 8447.975838] Lustre: server umount lustre-OST0000 complete [ 8459.199327] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 8471.879899] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 8511.450553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8511.819867] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8511.857618] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8511.955502] Lustre: lustre-MDT0000: new disk, initializing [ 8512.021989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8512.037794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8516.956616] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8529.167988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8529.330137] Lustre: 197338:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8529.412543] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8529.414462] Lustre: Skipped 1 previous similar message [ 8529.419723] LustreError: 198121:0:(llog.c:1696:llog_backup()) MGC192.168.202.123@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8529.430486] LustreError: 198121:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8529.446185] Lustre: 198121:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.123@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8529.487447] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8529.493880] Lustre: Skipped 1 previous similar message [ 8529.593251] Lustre: lustre-MDT0001: new disk, initializing [ 8529.789676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8529.888580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8529.902602] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8536.450957] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8547.580097] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8553.874804] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8561.481053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8561.855428] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8561.859867] Lustre: Skipped 1 previous similar message [ 8561.870893] LustreError: 199283:0:(llog.c:1696:llog_backup()) MGC192.168.202.123@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8561.882569] LustreError: 199283:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8561.891251] Lustre: 199283:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.123@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8561.908346] Lustre: 199283:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8561.940741] Lustre: lustre-OST0000: new disk, initializing [ 8561.945334] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8561.964838] Lustre: 199307:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8562.074117] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8569.473470] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8570.455270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8570.474467] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8570.625128] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8580.440212] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8586.156159] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8586.593878] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8591.089656] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8591.339854] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8594.412295] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8594.432522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8594.432949] Lustre: Skipped 2 previous similar messages [ 8594.443314] Lustre: Skipped 1 previous similar message [ 8599.535984] LustreError: 199290:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8599.552398] LustreError: 199290:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 8599.835704] Lustre: server umount lustre-OST0000 complete [ 8604.641577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8604.663500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8604.672359] Lustre: Skipped 1 previous similar message [ 8609.762376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8609.772932] Lustre: Skipped 1 previous similar message [ 8610.800451] Lustre: server umount lustre-MDT0000 complete [ 8614.884311] LustreError: 197345:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8614.913292] LustreError: 197345:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 8615.243304] LustreError: 197329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867359 with bad export cookie 621702579802976185 [ 8615.254237] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8615.256649] LustreError: 197329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8615.657931] Lustre: server umount lustre-MDT0001 complete [ 8624.240669] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 17:49:26 (1787867366) [ 8644.108359] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 8654.996365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8655.564859] LustreError: 202435:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8655.612862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8659.564182] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8668.251864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8673.899367] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8676.839328] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8683.710162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8691.035126] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8701.902849] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8702.157501] Lustre: lustre-OST0001: new disk, initializing [ 8702.160754] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8702.172796] Lustre: 204750:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8702.365110] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8702.379568] Lustre: Skipped 2 previous similar messages [ 8710.699974] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8711.290287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8711.310993] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8711.425364] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8722.771847] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8729.599924] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8731.690169] Lustre: DEBUG MARKER: ADD OST3 [ 8737.377478] loop: module loaded [ 8737.470254] loop0: detected capacity change from 0 to 204800000 [ 8737.847694] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8737.862126] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8737.875574] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8745.094311] Lustre: DEBUG MARKER: STOP OST3 [ 8745.152300] loop0: detected capacity change from 0 to 204800000 [ 8753.212835] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8753.537822] Lustre: lustre-OST0003: new disk, initializing [ 8753.549168] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8753.564920] Lustre: 206806:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8754.769636] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8754.969264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8755.000950] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8755.044430] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8756.833044] LustreError: 206890:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8757.142921] Lustre: Failing over lustre-OST0003 [ 8757.363879] Lustre: server umount lustre-OST0003 complete [ 8760.132843] LustreError: 203921:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8760.163981] LustreError: 203921:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [ 8760.300398] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8760.321115] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8760.350963] Lustre: Skipped 2 previous similar messages [ 8763.585949] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8776.657775] Lustre: DEBUG MARKER: START OST3 again [ 8785.718298] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8785.891325] Lustre: lustre-OST0003: Not available for connect from 0@lo (not set up) [ 8786.121442] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8786.136408] Lustre: Skipped 1 previous similar message [ 8786.162481] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8787.367885] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8787.566799] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8787.571964] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8794.050619] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8818.369897] Lustre: Failing over lustre-OST0003 [ 8818.626920] Lustre: server umount lustre-OST0003 complete [ 8818.662105] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8818.679284] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8818.680158] LustreError: Skipped 1 previous similar message [ 8818.684146] LustreError: 205133:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8818.684171] LustreError: 205133:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 15 previous similar messages [ 8818.803660] Lustre: Skipped 1 previous similar message [ 8837.603551] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8837.611501] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8837.640775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8837.645024] Lustre: Skipped 1 previous similar message [ 8844.097290] Lustre: server umount lustre-MDT0000 complete [ 8848.614077] LustreError: 202416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867592 with bad export cookie 621702579802977067 [ 8848.615203] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8848.639535] LustreError: 202416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8853.922104] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8853.948150] Lustre: Skipped 7 previous similar messages [ 8855.319102] Lustre: server umount lustre-MDT0001 complete [ 8859.880563] Lustre: server umount lustre-OST0000 complete [ 8865.938890] Lustre: server umount lustre-OST0001 complete [ 8875.994333] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 8888.639238] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 8938.332904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8938.585330] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8938.622935] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8938.749759] Lustre: lustre-MDT0000: new disk, initializing [ 8938.853956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8938.893966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8944.458503] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8955.739475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8955.884195] Lustre: 212374:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8955.972439] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8955.981313] Lustre: Skipped 1 previous similar message [ 8956.097333] Lustre: lustre-MDT0001: new disk, initializing [ 8956.212063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8956.226791] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8961.996471] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8972.348723] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8979.125808] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8986.291795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8986.742521] Lustre: 214341:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8988.427632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8988.433736] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8988.536387] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8994.852180] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9005.876784] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9011.925090] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9012.159678] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9017.452988] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9017.689744] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9024.484367] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9024.492658] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9024.515800] Lustre: Skipped 5 previous similar messages [ 9024.519318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9024.551921] LustreError: Skipped 1 previous similar message [ 9026.772954] Lustre: server umount lustre-OST0000 complete [ 9033.190194] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9038.094620] Lustre: server umount lustre-MDT0000 complete [ 9039.844495] LustreError: 212381:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9039.873645] LustreError: 212381:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 20 previous similar messages [ 9042.296541] LustreError: 212366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867786 with bad export cookie 621702579802979356 [ 9042.302596] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9042.309716] LustreError: 212366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9042.854097] Lustre: server umount lustre-MDT0001 complete [ 9047.609855] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 17:56:29 (1787867789) [ 9067.067700] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 9079.213248] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 9132.627697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9133.058319] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9133.097284] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9133.102025] Lustre: Skipped 1 previous similar message [ 9133.171690] Lustre: lustre-MDT0000: new disk, initializing [ 9133.174014] Lustre: Skipped 1 previous similar message [ 9133.380285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9139.566549] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9153.629249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9153.893163] Lustre: 218954:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9154.413432] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9160.638257] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9172.712670] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9181.729247] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9190.539234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9191.003292] Lustre: 220930:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9192.563569] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9192.814578] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9200.089247] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9211.572858] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9217.301907] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9217.680459] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9223.211671] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9223.445520] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9228.775182] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9228.785782] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9228.813119] Lustre: Skipped 3 previous similar messages [ 9228.818595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9228.824017] Lustre: Skipped 4 previous similar messages [ 9231.878440] Lustre: server umount lustre-OST0000 complete [ 9243.114113] Lustre: server umount lustre-MDT0000 complete [ 9244.139711] LustreError: 219816:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9244.172300] LustreError: 219816:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [ 9248.112354] LustreError: 218947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867992 with bad export cookie 621702579802980238 [ 9248.120470] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9248.858700] Lustre: server umount lustre-MDT0001 complete [ 9259.949085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9260.400624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9260.406590] Lustre: Skipped 5 previous similar messages [ 9266.314584] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9277.193112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9282.769376] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9292.880464] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9300.398500] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9308.558315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9316.747917] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9326.132912] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9342.946761] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9342.955791] LustreError: Skipped 1 previous similar message [ 9342.965195] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9342.984205] Lustre: Skipped 3 previous similar messages [ 9348.849383] Lustre: server umount lustre-OST0000 complete [ 9359.696190] Lustre: server umount lustre-MDT0000 complete [ 9363.690062] LustreError: 226258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868107 with bad export cookie 621702579802981120 [ 9363.691721] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9363.706627] LustreError: 226258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9364.210303] Lustre: server umount lustre-MDT0001 complete [ 9465.340789] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 9468.176470] Key type lgssc unregistered [ 9468.501529] LNet: 226847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9468.510939] LNetError: 226847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9468.526977] LNet: Removed LNI 192.168.202.123@tcp [ 9469.473208] Key type .llcrypt unregistered [ 9469.475084] Key type ._llcrypt unregistered [ 9497.626824] Key type ._llcrypt registered [ 9497.629661] Key type .llcrypt registered [ 9497.725425] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 9515.498410] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 9516.845730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9517.156976] alg: No test for adler32 (adler32-zlib) [ 9518.396709] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [ 9518.716433] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 9520.496710] Key type lgssc registered [ 9521.851584] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9563.034766] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9563.081043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9564.530854] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9564.620092] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9564.875750] Lustre: lustre-MDT0000: new disk, initializing [ 9565.056602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9565.085877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9571.299911] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9585.436569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9585.585092] Lustre: 230200:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9585.685536] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9585.690504] Lustre: Skipped 1 previous similar message [ 9585.789474] Lustre: lustre-MDT0001: new disk, initializing [ 9585.983845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9586.054944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9586.084366] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9592.302833] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9603.914707] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9611.446978] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9618.494081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9618.847347] Lustre: lustre-OST0000: new disk, initializing [ 9618.850326] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9618.856981] Lustre: 232172:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9618.919343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9626.256968] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9626.750119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9626.766811] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9626.851722] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9636.501326] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9641.292186] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9641.507517] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9646.663722] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9646.851806] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9652.723480] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9652.726897] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9652.741710] LustreError: Skipped 1 previous similar message [ 9652.750965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9652.762291] Lustre: Skipped 1 previous similar message [ 9652.787921] Lustre: Skipped 1 previous similar message [ 9654.842699] Lustre: server umount lustre-OST0000 complete [ 9662.948812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9662.953725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9662.962316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9662.983598] Lustre: Skipped 1 previous similar message [ 9667.698466] Lustre: server umount lustre-MDT0000 complete [ 9672.828432] LustreError: 230195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868416 with bad export cookie 6308911039813703633 [ 9672.846982] LustreError: 230195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9672.866179] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9673.286956] Lustre: server umount lustre-MDT0001 complete [ 9696.986696] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 9709.593766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9710.142687] LustreError: 235108:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9710.161019] LustreError: 235108:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 5 previous similar messages [ 9710.178133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9715.169923] LustreError: 235109:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9715.785627] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9720.293774] LustreError: 235108:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9725.414552] LustreError: 235109:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9725.910856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9726.209020] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9731.756533] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9735.005351] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9742.461253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9742.934930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9749.771441] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9759.090557] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9759.294564] Lustre: lustre-OST0001: new disk, initializing [ 9759.303553] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9759.308495] Lustre: 237416:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9759.405959] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9765.333512] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9769.112372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9769.124938] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9769.336152] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9777.899930] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9786.644830] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9803.979377] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 18:09:05 (1787868545) [ 9808.357157] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9808.369522] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9808.397305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9808.404526] Lustre: Skipped 1 previous similar message [ 9810.404366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9810.421764] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9810.447613] Lustre: Skipped 1 previous similar message [ 9814.578646] Lustre: server umount lustre-MDT0000 complete [ 9815.529459] LustreError: 235108:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9815.556260] LustreError: 235108:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 3 previous similar messages [ 9818.941409] LustreError: 236617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868562 with bad export cookie 6308911039813704515 [ 9818.942939] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9818.962055] LustreError: 236617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9819.780902] Lustre: server umount lustre-MDT0001 complete [ 9834.817376] Lustre: server umount lustre-OST0000 complete [ 9836.843636] Lustre: 228427:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787868564/real 1787868564] req@ffff9aa0c34f0000 x1874715750811008/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787868580 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9836.900784] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9836.931542] Lustre: Skipped 1 previous similar message [ 9840.305960] Lustre: server umount lustre-OST0001 complete [ 9848.960526] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 9859.975878] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 9900.298084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9900.546274] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9900.568385] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9900.628196] Lustre: lustre-MDT0000: new disk, initializing [ 9900.709517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9900.725884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9905.159216] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9915.509693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9915.665367] Lustre: 241361:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9915.728637] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9915.732664] Lustre: Skipped 1 previous similar message [ 9915.895801] Lustre: lustre-MDT0001: new disk, initializing [ 9916.109842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9916.136866] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9920.427196] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9930.073301] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9935.674677] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9943.022123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9943.256739] Lustre: lustre-OST0000: new disk, initializing [ 9943.259608] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9943.263147] Lustre: 243331:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9945.314252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9945.323660] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9945.499975] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9950.578401] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9962.435473] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9969.185873] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9969.453965] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9975.889275] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9976.216632] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9981.409391] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9981.421425] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9981.436138] LustreError: Skipped 1 previous similar message [ 9981.441813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9981.441822] Lustre: Skipped 3 previous similar messages [ 9981.492672] Lustre: Skipped 1 previous similar message [ 9984.763747] Lustre: server umount lustre-OST0000 complete [ 9991.675601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9991.687033] Lustre: Skipped 1 previous similar message [ 9996.046764] Lustre: server umount lustre-MDT0000 complete [ 9996.781453] LustreError: 241373:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9996.804837] LustreError: 241373:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [10000.491610] LustreError: 241356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868744 with bad export cookie 6308911039813706650 [10000.493193] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10000.502420] LustreError: 241356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10000.790377] Lustre: server umount lustre-MDT0001 complete [10009.132062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10009.569470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10009.579461] Lustre: Skipped 2 previous similar messages [10014.218565] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10014.690433] LustreError: 245550:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10014.721131] LustreError: 245550:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [10023.944949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10029.819409] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10038.293332] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10044.951554] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10052.598146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10059.725047] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10069.163674] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10217.957309] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10217.968339] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10217.984750] Lustre: Skipped 2 previous similar messages [10217.991345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10218.005214] Lustre: Skipped 2 previous similar messages [10222.754253] Lustre: server umount lustre-OST0000 complete [10229.217444] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10229.227773] LustreError: Skipped 1 previous similar message [10234.236045] Lustre: server umount lustre-MDT0000 complete [10236.386397] LustreError: 245550:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10236.416459] LustreError: 245550:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [10238.589590] LustreError: 245531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868982 with bad export cookie 6308911039813707532 [10238.603171] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10239.039799] Lustre: server umount lustre-MDT0001 complete [10337.160522] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [10340.683691] Key type lgssc unregistered [10341.186600] LNet: 249161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10341.190363] LNetError: 249161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10341.213284] LNet: Removed LNI 192.168.202.123@tcp [10342.377244] Key type .llcrypt unregistered [10342.379382] Key type ._llcrypt unregistered [10370.302432] Key type ._llcrypt registered [10370.304367] Key type .llcrypt registered [10370.424575] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [10389.253101] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [10390.394541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10390.459441] alg: No test for adler32 (adler32-zlib) [10391.491361] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [10391.754884] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [10393.416187] Key type lgssc registered [10394.671660] Lustre: Echo OBD driver; http://www.lustre.org/ [10450.616599] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [10463.340308] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10463.383499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10464.863788] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10464.956333] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10465.096334] Lustre: lustre-MDT0000: new disk, initializing [10465.249184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10465.271663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10469.762388] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10481.571155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10481.705736] Lustre: 253254:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10481.760610] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10481.765311] Lustre: Skipped 1 previous similar message [10481.857237] Lustre: lustre-MDT0001: new disk, initializing [10481.980825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10482.013826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10482.031324] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10486.860637] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10491.724595] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10499.534663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10499.890752] Lustre: lustre-OST0000: new disk, initializing [10499.905195] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10499.915273] Lustre: 254883:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10500.049597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10500.339790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10500.362872] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10500.616491] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10507.467396] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10520.037583] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10520.227215] Lustre: lustre-OST0001: new disk, initializing [10520.231773] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10520.239689] Lustre: 255756:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10520.327855] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10526.326809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10526.342621] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10526.479603] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10527.649644] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10540.122624] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10547.705717] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10562.530165] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10562.544398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10562.548095] Lustre: Skipped 2 previous similar messages [10566.388539] Lustre: server umount lustre-MDT0000 complete [10570.544658] LustreError: 253247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869314 with bad export cookie 8396026359544157365 [10570.554833] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10570.558279] LustreError: 253247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10571.004052] Lustre: server umount lustre-MDT0001 complete [10585.781375] Lustre: server umount lustre-OST0000 complete [10588.640245] Lustre: 250740:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787869316/real 1787869316] req@ffff9aa0cc184f00 x1874716666180352/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787869332 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10588.682899] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10588.690161] Lustre: Skipped 3 previous similar messages [10590.496688] Lustre: server umount lustre-OST0001 complete [10605.932183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10618.266767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10629.214914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10639.220365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10659.886816] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [10671.870353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10671.888405] Lustre: lustre-MDT0000: reset Object Index mappings [10672.251525] LustreError: 260769:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10672.269604] LustreError: 260769:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 5 previous similar messages [10672.337784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10676.952457] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10677.730434] LustreError: 260770:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10682.851812] LustreError: 260769:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10686.228972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10686.255288] Lustre: lustre-MDT0001: reset Object Index mappings [10686.857702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10691.695977] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10694.799776] Lustre: 261913:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10702.136669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10710.142950] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10711.795080] LustreError: 262267:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10717.175198] LustreError: 262703:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10718.203328] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10719.980689] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10720.305827] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10720.309966] Lustre: Skipped 1 previous similar message [10725.413626] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10727.639728] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10735.831360] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10739.966944] Lustre: 263783:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10745.825495] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10745.840728] Lustre: Skipped 1 previous similar message [10745.842800] LustreError: 260766:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10745.860947] LustreError: 260766:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 3 previous similar messages [10745.907974] Lustre: server umount lustre-MDT0000 complete [10750.175799] LustreError: 263097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869494 with bad export cookie 8396026359544164148 [10750.177045] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10750.186701] LustreError: 263097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10750.537861] Lustre: server umount lustre-MDT0001 complete [10766.053689] Lustre: server umount lustre-OST0000 complete [10766.945124] Lustre: 250739:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787869494/real 1787869494] req@ffff9aa0cf30f0c0 x1874716666229120/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787869510 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10766.984323] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10767.000667] Lustre: Skipped 2 previous similar messages [10770.021691] Lustre: server umount lustre-OST0001 complete [10790.941351] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 18:25:32 (1787869532) [10810.272437] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [10822.589650] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [10862.783635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10862.995651] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10863.017601] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10863.076594] Lustre: lustre-MDT0000: new disk, initializing [10863.136132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10863.154620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10867.519353] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10877.874077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10878.014130] Lustre: 268040:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10878.102354] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10878.119454] Lustre: Skipped 1 previous similar message [10878.268133] Lustre: lustre-MDT0001: new disk, initializing [10878.512071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10878.523586] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10883.236610] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10893.105549] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10900.012745] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10907.812254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10908.236407] Lustre: lustre-OST0000: new disk, initializing [10908.246522] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10908.265348] Lustre: 270007:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10909.838606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10909.854170] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10910.026047] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10916.367093] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10926.914123] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10932.159596] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10932.424666] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10937.267509] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10937.550420] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10940.901412] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10940.909225] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10940.920571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10940.937668] Lustre: Skipped 1 previous similar message [10940.963226] Lustre: Skipped 2 previous similar messages [10946.029177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10946.040172] Lustre: Skipped 1 previous similar message [10946.386846] Lustre: server umount lustre-OST0000 complete [10955.242835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10955.249833] LustreError: Skipped 1 previous similar message [10955.255052] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10955.272972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10957.564217] Lustre: server umount lustre-MDT0000 complete [10961.386658] LustreError: 269628:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10961.401736] LustreError: 269628:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 1 previous similar message [10963.005830] LustreError: 268835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869706 with bad export cookie 8396026359544166248 [10963.015661] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10963.023527] LustreError: 268835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10963.486226] Lustre: server umount lustre-MDT0001 complete [10986.036630] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [10998.973369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10999.323830] LustreError: 272946:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10999.389837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10999.394889] Lustre: Skipped 2 previous similar messages [11005.248903] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11014.642588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11021.158444] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11024.105248] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11031.616829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11038.164950] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11047.694509] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11047.932891] Lustre: lustre-OST0001: new disk, initializing [11047.937521] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11047.945334] Lustre: 275254:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11049.202833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11049.217856] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11049.341406] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11054.286264] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11064.537099] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11068.771348] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11074.478781] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [11074.503296] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11081.700078] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11081.705748] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11081.740429] Lustre: Skipped 1 previous similar message [11081.751646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11081.764734] Lustre: Skipped 2 previous similar messages [11086.997681] Lustre: server umount lustre-MDT0000 complete [11090.404948] LustreError: 272947:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11090.424720] LustreError: 272947:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 6 previous similar messages [11090.833353] LustreError: 272928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869834 with bad export cookie 8396026359544167130 [11090.840252] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11090.846370] LustreError: 272928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11091.075919] Lustre: server umount lustre-MDT0001 complete [11104.934794] Lustre: server umount lustre-OST0000 complete [11119.756888] Lustre: server umount lustre-OST0001 complete [11137.682336] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [11147.565841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11147.895642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11147.899994] Lustre: Skipped 3 previous similar messages [11152.575832] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11161.488218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11167.030864] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11169.674491] Lustre: 279454:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11177.145240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11186.044721] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11192.905958] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11195.378530] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11201.065214] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11201.611872] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11208.348482] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11211.401734] Lustre: 281328:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11215.939986] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11216.173522] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11225.687257] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 18:32:47 (1787869967) [11241.955763] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11241.964049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11241.968253] Lustre: Skipped 5 previous similar messages [11241.980224] Lustre: Skipped 7 previous similar messages [11247.094544] LustreError: 278310:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11247.115808] LustreError: 278310:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 9 previous similar messages [11247.333665] Lustre: server umount lustre-MDT0000 complete [11251.391176] LustreError: 281330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869995 with bad export cookie 8396026359544173675 [11251.393875] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11251.409257] LustreError: 281330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11257.882912] Lustre: server umount lustre-MDT0001 complete [11276.258978] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11276.575257] Lustre: server umount lustre-OST0000 complete [11295.200159] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11295.546500] Lustre: server umount lustre-OST0001 complete [11304.979615] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [11317.347693] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [11363.987476] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [11375.114945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11375.376182] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11375.410588] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11375.542699] Lustre: lustre-MDT0000: new disk, initializing [11375.651885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11379.701374] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11389.690252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11389.819375] Lustre: 285576:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11390.102897] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11394.932099] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11407.285822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11407.676578] Lustre: 287209:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11407.840990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11407.857346] Lustre: Skipped 5 previous similar messages [11409.628148] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11414.486480] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11426.681434] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11426.967339] Lustre: 288080:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11428.640346] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11428.651342] Lustre: Skipped 1 previous similar message [11428.808202] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11432.835727] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11444.136715] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11448.267803] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11448.274223] Lustre: Skipped 1 previous similar message [11466.876927] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 18:36:48 (1787870208) [11469.803139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11469.816480] Lustre: Skipped 3 previous similar messages [11469.818892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11469.831784] Lustre: Skipped 5 previous similar messages [11471.405267] Lustre: server umount lustre-MDT0000 complete [11474.586645] LustreError: 286380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870218 with bad export cookie 8396026359544181340 [11474.594432] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11474.596612] LustreError: 286380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11474.957486] Lustre: server umount lustre-MDT0001 complete [11489.191560] Lustre: server umount lustre-OST0000 complete [11490.272204] Lustre: 250741:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787870218/real 1787870218] req@ffff9aa0c2a2f0c0 x1874716666491008/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787870234 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11493.260799] Lustre: server umount lustre-OST0001 complete [11503.729886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11504.639088] LustreError: 290328:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11504.681898] LustreError: 290328:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 5 previous similar messages [11510.283254] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11520.270388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11526.249380] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11535.321878] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11541.778316] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11549.215045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11555.723880] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11563.543939] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11584.888195] loop1: detected capacity change from 0 to 204800000 [11585.096156] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11585.103634] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11585.121045] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11591.048545] loop1: detected capacity change from 0 to 204800000 [11596.255483] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11596.337855] LustreError: 290317:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs12758: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11596.349512] LustreError: 290317:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs12758 [11596.353696] LustreError: 290317:0:(mgs_handler.c:517:mgs_target_reg()) fs12758-MDT0001: Error checking target index: rc = -13 [11596.361381] LustreError: fs12758-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11596.372345] LustreError: 294047:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11596.377785] LustreError: 294047:0:(tgt_mount.c:2129:server_put_super()) no obd fs12758-MDT0001 [11596.383582] LustreError: 294047:0:(tgt_mount.c:116:server_deregister_mount()) fs12758-MDT0001 not registered [11596.414428] Lustre: server umount fs12758-MDT0001 complete [11596.418505] LustreError: 294047:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11606.009738] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11606.266813] Lustre: Setting parameter fs12758-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs12758-MDT0001 [11606.339245] Lustre: srv-fs12758-MDT0001: No data found on store. Initialize space. [11606.349376] Lustre: Skipped 4 previous similar messages [11606.463923] Lustre: fs12758-MDT0001: new disk, initializing [11606.468276] Lustre: Skipped 3 previous similar messages [11606.622453] Lustre: srv-fs12758-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11606.636289] Lustre: 294514:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs12758-MDT0001-mdtlov: can't create new llog: rc = -115 [11606.641784] LustreError: 294514:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs12758-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11611.253526] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11618.196465] Lustre: server umount fs12758-MDT0001 complete [11632.619501] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11633.323672] loop1: detected capacity change from 0 to 204800000 [11639.695765] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11640.288391] Lustre: 296083:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs12758-MDT0001-mdtlov: can't create new llog: rc = -115 [11640.291619] LustreError: 296083:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs12758-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11645.118383] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11647.995107] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11650.279257] Lustre: server umount fs12758-MDT0001 complete [11661.639258] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.23@tcp (stopping) [11663.334382] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11663.349087] Lustre: Skipped 5 previous similar messages [11666.305270] Lustre: server umount lustre-MDT0000 complete [11675.735261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11675.964294] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11681.793818] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11683.568533] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11693.006933] loop1: detected capacity change from 0 to 204800000 [11698.309661] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11698.594570] Lustre: 298530:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs12758-MDT0001-mdtlov: can't create new llog: rc = -115 [11698.603818] LustreError: 298530:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs12758-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11703.006557] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11707.875744] Lustre: server umount fs12758-MDT0001 complete [11732.982265] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11732.987394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11732.990288] Lustre: Skipped 4 previous similar messages [11737.041506] Lustre: server umount lustre-OST0000 complete [11747.101528] Lustre: server umount lustre-MDT0000 complete [11750.434739] LustreError: 290308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870494 with bad export cookie 8396026359544184021 [11750.457366] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11750.783908] Lustre: server umount lustre-MDT0001 complete [11761.957108] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [11764.591685] Key type lgssc unregistered [11764.867488] LNet: 300876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11764.880303] LNetError: 300876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11764.905830] LNet: Removed LNI 192.168.202.123@tcp [11765.685282] Key type .llcrypt unregistered [11765.687283] Key type ._llcrypt unregistered [11786.801700] Key type ._llcrypt registered [11786.807955] Key type .llcrypt registered [11786.971983] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [11788.353968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11788.439481] alg: No test for adler32 (adler32-zlib) [11789.710212] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [11790.075813] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [11791.832358] Key type lgssc registered [11793.204492] Lustre: Echo OBD driver; http://www.lustre.org/ [11800.885939] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11800.901307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11802.434245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11807.207706] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11812.841493] LustreError: 302150:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11812.850638] LustreError: 302150:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [11816.309393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11816.888927] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11821.425085] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11833.392815] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11840.976812] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11850.760849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11851.295808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11857.226632] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11861.553120] LustreError: 303979:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11865.681358] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11866.624507] LustreError: 303981:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11871.726504] LustreError: 303979:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11877.687393] LustreError: 303980:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11877.699583] LustreError: 303980:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [11887.887625] LustreError: 303979:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11887.915177] LustreError: 303979:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 5 previous similar messages [11895.660804] loop1: detected capacity change from 0 to 204800000 [11895.853446] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11895.864487] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11895.875566] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11902.087771] loop1: detected capacity change from 0 to 204800000 [11908.361235] LustreError: 303981:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11908.396375] LustreError: 303981:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 11 previous similar messages [11908.651753] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11908.806481] Lustre: fs15929-OST0000: new disk, initializing [11908.809304] Lustre: srv-fs15929-OST0000: No data found on store. Initialize space. [11908.822349] Lustre: 305907:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11908.952295] Lustre: fs15929-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11915.485488] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11923.719305] Lustre: server umount fs15929-OST0000 complete [11936.171078] loop1: detected capacity change from 0 to 204800000 [11939.335259] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11942.655487] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11942.856613] Lustre: fs15929-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11944.200269] LustreError: 303980:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11944.227954] LustreError: 303980:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 20 previous similar messages [11949.556623] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11950.021426] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11958.898041] loop2: detected capacity change from 0 to 204800000 [11959.151740] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11959.174291] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11959.207890] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11964.557763] loop2: detected capacity change from 0 to 204800000 [11971.810717] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11971.906844] LustreError: fs15929-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11971.917674] LustreError: fs15929-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11971.926601] LustreError: 308950:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11971.932151] LustreError: 308950:0:(tgt_mount.c:2129:server_put_super()) no obd fs15929-OST0001 [11971.936864] LustreError: 308950:0:(tgt_mount.c:116:server_deregister_mount()) fs15929-OST0001 not registered [11971.961257] Lustre: server umount fs15929-OST0001 complete [11971.963561] LustreError: 308950:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11982.904740] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11983.036884] Lustre: fs15929-OST0001: new disk, initializing [11983.040274] Lustre: srv-fs15929-OST0001: No data found on store. Initialize space. [11983.048579] Lustre: 309501:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11983.135251] Lustre: fs15929-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11990.463848] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11998.913483] Lustre: server umount fs15929-OST0000 complete [12008.691746] Lustre: server umount fs15929-OST0001 complete [12010.769994] LustreError: 307180:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12010.796674] LustreError: 307180:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 38 previous similar messages [12021.313267] loop1: detected capacity change from 0 to 204800000 [12027.008128] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12027.256778] Lustre: fs15929-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12035.137664] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12041.458458] loop2: detected capacity change from 0 to 204800000 [12047.149028] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12053.166178] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12061.700459] Lustre: server umount fs15929-OST0000 complete [12070.470747] Lustre: server umount fs15929-OST0001 complete [12086.753403] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12086.760129] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12086.777237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12087.800897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12087.801397] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12088.981376] Lustre: server umount lustre-OST0000 complete [12092.899491] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12092.900601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12092.906420] Lustre: Skipped 1 previous similar message [12092.913522] Lustre: Skipped 1 previous similar message [12098.018209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12098.030069] Lustre: Skipped 2 previous similar messages [12098.956784] Lustre: server umount lustre-MDT0000 complete [12102.152571] LustreError: 302130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870846 with bad export cookie 938074201997397072 [12102.158696] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12102.162671] LustreError: 302130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12102.554652] Lustre: server umount lustre-MDT0001 complete [12114.318740] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [12117.004790] Key type lgssc unregistered [12117.299959] LNet: 315424:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12117.312449] LNetError: 315424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12117.331311] LNet: Removed LNI 192.168.202.123@tcp [12118.305261] Key type .llcrypt unregistered [12118.309382] Key type ._llcrypt unregistered [12152.485210] Key type ._llcrypt registered [12152.486895] Key type .llcrypt registered [12152.742644] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [12177.753540] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [12178.761538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12178.952384] alg: No test for adler32 (adler32-zlib) [12180.130759] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [12180.365387] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [12182.016364] Key type lgssc registered [12183.524077] Lustre: Echo OBD driver; http://www.lustre.org/ [12219.736839] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12219.787398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12221.053727] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12221.095946] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12221.198769] Lustre: lustre-MDT0000: new disk, initializing [12221.342306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12221.360413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12225.407480] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12235.679287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12235.852219] Lustre: 319289:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12235.929872] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12235.934448] Lustre: Skipped 1 previous similar message [12236.041554] Lustre: lustre-MDT0001: new disk, initializing [12236.213515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12236.250557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12236.266259] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12240.927936] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12250.963688] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12256.670613] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12262.855463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12263.102939] Lustre: lustre-OST0000: new disk, initializing [12263.111805] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12263.120495] Lustre: 321257:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12263.227391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12268.617778] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12272.872594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12272.889724] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12272.925484] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12277.363777] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12279.668652] Lustre: Mounted lustre-client - version 2.17.57_89_g2dd35b7 [12280.777655] Lustre: Unmounted lustre-client [12293.609470] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12293.613532] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12293.625708] LustreError: Skipped 1 previous similar message [12293.636262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12293.645172] Lustre: Skipped 1 previous similar message [12295.137697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12295.149996] Lustre: Skipped 1 previous similar message [12296.757863] Lustre: server umount lustre-OST0000 complete [12302.817676] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12302.827237] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12302.858413] Lustre: Skipped 1 previous similar message [12302.872314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12305.381792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12305.384138] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12307.552379] Lustre: server umount lustre-MDT0000 complete [12311.539051] LustreError: 319280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871055 with bad export cookie 9789051770395574520 [12311.550519] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12311.552140] LustreError: 319280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12311.932751] Lustre: server umount lustre-MDT0001 complete [12324.688139] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [12327.543492] Key type lgssc unregistered [12336.162739] LNet: 323420:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12336.181931] LNetError: 323420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12412.008372] LNet: Removed LNI 192.168.202.123@tcp [12413.606215] Key type .llcrypt unregistered [12413.613642] Key type ._llcrypt unregistered [12435.575708] Key type ._llcrypt registered [12435.579305] Key type .llcrypt registered [12435.678426] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [12453.619644] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [12455.133563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12455.356020] alg: No test for adler32 (adler32-zlib) [12456.521413] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [12456.790654] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [12458.552159] Key type lgssc registered [12460.053803] Lustre: Echo OBD driver; http://www.lustre.org/ [12499.577768] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 18:54:01 (1787871241) [12508.341929] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12508.380740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12509.802142] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12509.837747] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12509.948389] Lustre: lustre-MDT0000: new disk, initializing [12510.110089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12510.127306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12515.278611] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12528.082095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12528.292562] Lustre: 326647:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12528.360273] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12528.367528] Lustre: Skipped 1 previous similar message [12528.453844] Lustre: lustre-MDT0001: new disk, initializing [12528.582202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12528.640198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12528.655261] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12533.969157] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12544.063000] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12550.231910] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12559.003065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12559.487785] Lustre: lustre-OST0000: new disk, initializing [12559.499841] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12559.517629] Lustre: 328613:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12559.645176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12565.973438] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12569.202901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12569.232704] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12569.334666] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12576.342718] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12598.230694] Lustre: server umount lustre-OST0000 complete [12605.410109] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12605.414364] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12605.439823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12605.466112] Lustre: Skipped 1 previous similar message [12610.530090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12610.540935] Lustre: Skipped 2 previous similar messages [12612.815546] Lustre: server umount lustre-MDT0000 complete [12616.663917] Lustre: 324819:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787871344/real 1787871344] req@000000009ee8716a x1874718831449728/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787871360 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12616.665876] LustreError: 326637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871360 with bad export cookie 4074385220816496699 [12616.666925] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12616.743593] LustreError: 326637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12627.141862] Lustre: server umount lustre-MDT0001 complete [12639.439606] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [12642.250743] Key type lgssc unregistered [12642.551664] LNet: 330528:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12642.556592] LNetError: 330528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12642.582241] LNet: Removed LNI 192.168.202.123@tcp [12643.585900] Key type .llcrypt unregistered [12643.589546] Key type ._llcrypt unregistered [12669.884916] Key type ._llcrypt registered [12669.887292] Key type .llcrypt registered [12670.043834] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [12671.379485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12671.680421] alg: No test for adler32 (adler32-zlib) [12672.763973] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [12673.006485] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [12674.664838] Key type lgssc registered [12675.617700] Lustre: Echo OBD driver; http://www.lustre.org/ [12683.553942] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12683.568222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12684.938390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12688.992865] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12694.509552] LustreError: 331805:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12694.521637] LustreError: 331805:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [12697.173415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12697.541715] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12702.671658] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12710.215768] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12716.882646] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12724.828365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12725.295283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12732.135824] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12740.223934] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12751.550583] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12753.282613] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12759.010959] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12759.017970] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12759.028162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12761.058705] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12761.064526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12761.076840] Lustre: Skipped 1 previous similar message [12761.079972] Lustre: Skipped 1 previous similar message [12763.420034] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.23@tcp (stopping) [12763.434598] Lustre: Skipped 1 previous similar message [12764.947832] Lustre: server umount lustre-MDT0000 complete [12766.179334] LustreError: 333839:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12766.207206] LustreError: 333839:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [12768.517836] LustreError: 331800:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12769.160756] LustreError: 331784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871513 with bad export cookie 2131344229597934299 [12769.162573] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12769.181296] LustreError: 331784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12771.308796] LustreError: 331801:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12771.309412] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12771.375386] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12775.674313] Lustre: server umount lustre-MDT0001 complete [12780.152512] Lustre: server umount lustre-OST0000 complete [12790.219293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12795.701468] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12798.722391] Lustre: 335988:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.123@tcp [12801.166660] Lustre: server umount MGS complete [12809.598228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12809.892369] LustreError: 336559:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12809.913493] LustreError: 336559:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [12809.962132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12815.153946] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12818.502489] LustreError: 336556:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12818.535519] LustreError: 336556:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 2 previous similar messages [12825.152835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12825.597977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12830.774250] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12841.163444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12841.464029] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12848.190617] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12856.660569] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12858.144392] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12864.123269] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12865.916496] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12872.165606] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [12872.175147] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12872.199648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12872.202489] Lustre: Skipped 2 previous similar messages [12873.623147] Lustre: server umount lustre-OST0000 complete [12878.307787] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12878.328496] Lustre: Skipped 2 previous similar messages [12883.425449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12883.429332] Lustre: Skipped 4 previous similar messages [12884.102630] Lustre: server umount lustre-MDT0000 complete [12887.853038] LustreError: 336539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871631 with bad export cookie 2131344229597935139 [12887.860244] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12887.868326] LustreError: 336539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12888.203165] Lustre: server umount lustre-MDT0001 complete [12900.467850] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [12903.447315] Key type lgssc unregistered [12903.725189] LNet: 340000:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12903.732567] LNetError: 340000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12903.751929] LNet: Removed LNI 192.168.202.123@tcp [12904.710132] Key type .llcrypt unregistered [12904.712449] Key type ._llcrypt unregistered [12924.297823] Key type ._llcrypt registered [12924.300142] Key type .llcrypt registered [12924.397173] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [12940.731621] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [12942.175971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12942.369721] alg: No test for adler32 (adler32-zlib) [12943.525483] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [12943.802525] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [12945.576818] Key type lgssc registered [12946.940814] Lustre: Echo OBD driver; http://www.lustre.org/ [12989.542146] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12989.568557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12990.950746] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12991.024221] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12991.125101] Lustre: lustre-MDT0000: new disk, initializing [12991.211582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12991.236688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12997.237501] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [13010.004050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13010.150067] Lustre: 343186:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13010.274315] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13010.280529] Lustre: Skipped 1 previous similar message [13010.409927] Lustre: lustre-MDT0001: new disk, initializing [13010.588544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13010.661162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13010.682496] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13016.360624] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [13027.056500] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13033.302952] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13040.900170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13041.225336] Lustre: lustre-OST0000: new disk, initializing [13041.231273] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13041.240660] Lustre: 345155:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13041.336739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13046.858640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13046.877291] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13047.015368] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13048.754777] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [13059.626343] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13066.305459] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13066.580627] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13072.743532] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13073.083534] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13077.993660] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13077.995837] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13078.007675] LustreError: Skipped 1 previous similar message [13078.033996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13081.968142] Lustre: server umount lustre-OST0000 complete [13087.201126] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13087.216344] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13087.234667] Lustre: Skipped 1 previous similar message [13087.242955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13087.253803] Lustre: Skipped 1 previous similar message [13088.226794] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13092.605930] Lustre: server umount lustre-MDT0000 complete [13096.439878] LustreError: 345154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871840 with bad export cookie 6107172126302822428 [13096.443624] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13096.449268] LustreError: 345154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13096.709308] Lustre: server umount lustre-MDT0001 complete [13108.982636] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 19:04:10 (1787871850) [13128.923040] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [13144.790887] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [13183.303719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13183.548649] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13183.573538] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13183.639593] Lustre: lustre-MDT0000: new disk, initializing [13183.700830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13183.716240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13188.349870] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [13199.886748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13200.004664] Lustre: 349942:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13200.044619] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13200.048376] Lustre: Skipped 1 previous similar message [13200.106606] Lustre: lustre-MDT0001: new disk, initializing [13200.199908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13200.242497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13200.258832] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13204.830416] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [13215.421516] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13220.671583] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13229.165912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13229.615659] Lustre: lustre-OST0000: new disk, initializing [13229.621433] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13229.635700] Lustre: 351912:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13229.765474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13236.804793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13236.816557] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13237.019675] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13237.260672] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [13250.738664] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13298.657673] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13298.661745] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13298.674963] LustreError: Skipped 1 previous similar message [13298.679698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13298.679706] Lustre: Skipped 2 previous similar messages [13298.711586] Lustre: Skipped 1 previous similar message [13299.885344] Lustre: server umount lustre-OST0000 complete [13307.873864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13307.889708] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13307.916852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13307.927738] Lustre: Skipped 1 previous similar message [13310.489791] Lustre: server umount lustre-MDT0000 complete [13314.019412] LustreError: 349954:0:(ldlm_lib.c:1212:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13314.038919] LustreError: 349954:0:(ldlm_lib.c:1212:target_handle_connect()) Skipped 4 previous similar messages [13314.376283] LustreError: 349934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872058 with bad export cookie 6107172126302823310 [13314.386156] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13314.387644] LustreError: 349934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13314.756984] Lustre: server umount lustre-MDT0001 complete [13328.425983] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [13331.859601] Key type lgssc unregistered [13332.222477] LNet: 354090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13332.238421] LNetError: 354090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13332.256816] LNet: Removed LNI 192.168.202.123@tcp [13333.218987] Key type .llcrypt unregistered [13333.220995] Key type ._llcrypt unregistered [13437.340628] Key type ._llcrypt registered [13437.342730] Key type .llcrypt registered [13437.469579] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [13437.753381] Key type .llcrypt unregistered [13437.756071] Key type ._llcrypt unregistered [13524.934681] Key type ._llcrypt registered [13524.938703] Key type .llcrypt registered [13525.079955] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [13525.452515] Key type .llcrypt unregistered [13525.454052] Key type ._llcrypt unregistered [13556.282450] Key type ._llcrypt registered [13556.284591] Key type .llcrypt registered [13556.372764] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [13556.702430] Key type .llcrypt unregistered [13556.709653] Key type ._llcrypt unregistered [13590.147867] Key type ._llcrypt registered [13590.151278] Key type .llcrypt registered [13590.284645] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [13590.558756] Key type .llcrypt unregistered [13590.561434] Key type ._llcrypt unregistered [13675.511641] Key type ._llcrypt registered [13675.514121] Key type .llcrypt registered [13675.667895] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [13676.017335] Key type .llcrypt unregistered [13676.027434] Key type ._llcrypt unregistered [13709.624940] Key type ._llcrypt registered [13709.626024] Key type .llcrypt registered [13709.740541] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [13710.074404] Key type .llcrypt unregistered [13710.076775] Key type ._llcrypt unregistered [13741.826412] Key type ._llcrypt registered [13741.829883] Key type .llcrypt registered [13741.926720] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [13742.170266] Key type .llcrypt unregistered [13742.180459] Key type ._llcrypt unregistered [13781.501500] Key type ._llcrypt registered [13781.504355] Key type .llcrypt registered [13781.606583] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [13782.187297] Key type .llcrypt unregistered [13782.191461] Key type ._llcrypt unregistered [13821.066848] Key type ._llcrypt registered [13821.073431] Key type .llcrypt registered [13821.198639] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [13821.665405] Key type .llcrypt unregistered [13821.669136] Key type ._llcrypt unregistered [13853.960990] Key type ._llcrypt registered [13853.964458] Key type .llcrypt registered [13854.056452] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [13854.300076] Key type .llcrypt unregistered [13854.301627] Key type ._llcrypt unregistered [13925.787434] Key type ._llcrypt registered [13925.789284] Key type .llcrypt registered [13925.972442] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [13926.395305] Key type .llcrypt unregistered [13926.399130] Key type ._llcrypt unregistered [13958.861737] Key type ._llcrypt registered [13958.864778] Key type .llcrypt registered [13958.995463] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [13959.335466] Key type .llcrypt unregistered [13959.337088] Key type ._llcrypt unregistered [13992.756402] Key type ._llcrypt registered [13992.759306] Key type .llcrypt registered [13992.845886] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [13993.098305] Key type .llcrypt unregistered [13993.104508] Key type ._llcrypt unregistered [14045.274771] Key type ._llcrypt registered [14045.276434] Key type .llcrypt registered [14045.387475] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [14063.012814] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [14064.597838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14064.618453] alg: No test for adler32 (adler32-zlib) [14065.858941] Lustre: Lustre: Build Version: 2.17.57_89_g2dd35b7 [14066.177858] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [14067.888339] Key type lgssc registered [14069.218524] Lustre: Echo OBD driver; http://www.lustre.org/ [14106.342425] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14106.367233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14107.634361] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14107.666774] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14107.791376] Lustre: tst250fs-MDT0000: new disk, initializing [14107.943296] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14107.997259] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14112.896706] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [14125.068582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14125.255104] Lustre: 374488:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14125.327032] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14125.337645] Lustre: Skipped 1 previous similar message [14125.442935] Lustre: tst250fs-MDT0001: new disk, initializing [14125.567357] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14125.623692] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14125.638206] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14130.744602] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [14135.715294] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14143.037508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14143.413974] Lustre: tst250fs-OST0000: new disk, initializing [14143.424069] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14143.432220] Lustre: 376122:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14143.596304] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14143.846289] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14143.854625] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14144.018411] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14150.723653] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [14161.299251] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14161.469300] Lustre: tst250fs-OST0001: new disk, initializing [14161.472278] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14161.480922] Lustre: 376993:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14161.578738] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14167.156496] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14167.176857] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14167.345188] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14168.800376] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [14175.178754] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14178.045878] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [14178.057674] Lustre: Skipped 2 previous similar messages [14197.217545] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14197.226289] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14197.245404] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14198.244594] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14198.245842] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14203.798371] Lustre: server umount tst250fs-MDT0000 complete [14207.790181] LustreError: 377667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872951 with bad export cookie 9060818551454530195 [14207.792662] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14207.798370] LustreError: 377667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14208.051522] Lustre: server umount tst250fs-MDT0001 complete [14221.284177] Lustre: server umount tst250fs-OST0000 complete [14224.354230] Lustre: 372713:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787872952/real 1787872952] req@ffff9aa1fe91f840 x1874720519690112/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787872968 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14224.393219] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14224.415743] Lustre: Skipped 2 previous similar messages [14227.014670] Lustre: server umount tst250fs-OST0001 complete [14249.200416] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [14259.004365] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [14298.481115] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 19:24:00 (1787873040) [14300.535172] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14320.883327] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [14331.542956] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [14364.759291] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14021 sec ========= 19:25:06 (1787873106) [14366.765501] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:25:08 (1787873108) === [14370.831745] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:25:12 (1787873112) === [14413.318259] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [14416.669198] Key type lgssc unregistered [14417.051986] LNet: 386139:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14417.058220] LNetError: 386139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14417.077303] LNet: Removed LNI 192.168.202.123@tcp [14418.188341] Key type .llcrypt unregistered [14418.190292] Key type ._llcrypt unregistered