[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 549159339 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2560MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003178] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008525] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.010012] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.011012] pid_max: default: 32768 minimum: 301 [ 0.013039] LSM: Security Framework initializing [ 0.014055] Yama: becoming mindful. [ 0.015043] SELinux: Initializing. [ 0.016063] *** VALIDATE selinux *** [ 0.025059] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.030457] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.031148] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032104] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034112] *** VALIDATE tmpfs *** [ 0.035452] *** VALIDATE proc *** [ 0.036230] *** VALIDATE cgroup *** [ 0.037011] *** VALIDATE cgroup2 *** [ 0.038251] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039161] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041029] Spectre V2 : User space: Vulnerable [ 0.042009] Speculative Store Bypass: Vulnerable [ 0.045662] debug: unmapping init [mem 0xffffffffbc459000-0xffffffffbc460fff] [ 0.047910] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048700] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049022] ... version: 2 [ 0.050010] ... bit width: 48 [ 0.051010] ... generic registers: 4 [ 0.052012] ... value mask: 0000ffffffffffff [ 0.053014] ... max period: 00007fffffffffff [ 0.054013] ... fixed-purpose events: 3 [ 0.055011] ... event mask: 000000070000000f [ 0.056250] rcu: Hierarchical SRCU implementation. [ 0.058434] smp: Bringing up secondary CPUs ... [ 0.059607] x86: Booting SMP configuration: [ 0.060026] .... node #0, CPUs: #1 #2 #3 [ 0.063289] smp: Brought up 1 node, 4 CPUs [ 0.065011] smpboot: Max logical packages: 1 [ 0.066017] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.134417] node 0 deferred pages initialised in 66ms [ 0.139088] devtmpfs: initialized [ 0.141255] x86/mm: Memory block size: 128MB [ 0.144445] gcov: version magic: 0x41383552 [ 0.148247] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.149075] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.152326] pinctrl core: initialized pinctrl subsystem [ 0.154173] [ 0.154820] ************************************************************* [ 0.158013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.160010] ** ** [ 0.163013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.166013] ** ** [ 0.168009] ** This means that this kernel is built to expose internal ** [ 0.170010] ** IOMMU data structures, which may compromise security on ** [ 0.173012] ** your system. ** [ 0.175009] ** ** [ 0.177010] ** If you see this message and you are not debugging the ** [ 0.180013] ** kernel, report this immediately to your vendor! ** [ 0.182010] ** ** [ 0.184010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.187010] ************************************************************* [ 0.189490] NET: Registered protocol family 16 [ 0.191480] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.193046] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.195043] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.199102] cpuidle: using governor menu [ 0.202182] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.205581] PCI: Using configuration type 1 for base access [ 0.208124] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.220089] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.223019] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.228037] cryptd: max_cpu_qlen set to 1000 [ 0.232250] ACPI: Added _OSI(Module Device) [ 0.233011] ACPI: Added _OSI(Processor Device) [ 0.235011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.236010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.241085] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.246613] ACPI: Interpreter enabled [ 0.248051] ACPI: PM: (supports S0 S3 S4 S5) [ 0.250028] ACPI: Using IOAPIC for interrupt routing [ 0.252108] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.257411] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.269254] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.272028] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.278019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.282070] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.287301] acpiphp: Slot [2] registered [ 0.289109] acpiphp: Slot [5] registered [ 0.290083] acpiphp: Slot [6] registered [ 0.292086] acpiphp: Slot [7] registered [ 0.294109] acpiphp: Slot [8] registered [ 0.296094] acpiphp: Slot [9] registered [ 0.297092] acpiphp: Slot [10] registered [ 0.299148] acpiphp: Slot [3] registered [ 0.301072] acpiphp: Slot [4] registered [ 0.302076] acpiphp: Slot [11] registered [ 0.304084] acpiphp: Slot [12] registered [ 0.306103] acpiphp: Slot [13] registered [ 0.307081] acpiphp: Slot [14] registered [ 0.309106] acpiphp: Slot [15] registered [ 0.311072] acpiphp: Slot [16] registered [ 0.312088] acpiphp: Slot [17] registered [ 0.314078] acpiphp: Slot [18] registered [ 0.316068] acpiphp: Slot [19] registered [ 0.318093] acpiphp: Slot [20] registered [ 0.319101] acpiphp: Slot [21] registered [ 0.321081] acpiphp: Slot [22] registered [ 0.323099] acpiphp: Slot [23] registered [ 0.327111] acpiphp: Slot [24] registered [ 0.329080] acpiphp: Slot [25] registered [ 0.330076] acpiphp: Slot [26] registered [ 0.332157] acpiphp: Slot [27] registered [ 0.333090] acpiphp: Slot [28] registered [ 0.335085] acpiphp: Slot [29] registered [ 0.337121] acpiphp: Slot [30] registered [ 0.339090] acpiphp: Slot [31] registered [ 0.341056] PCI host bridge to bus 0000:00 [ 0.342013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.345017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.348019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.351022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.354020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.357022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.360235] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.363009] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.367253] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.377013] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.382534] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.386016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.388015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.391015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.395345] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.398840] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.401036] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.404880] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.410024] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.424011] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.430013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.434643] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.474013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.484013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.507012] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.522369] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.536012] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.545011] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.567012] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.579816] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.600013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.632013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.669013] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.683662] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.698014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.709013] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.745014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.766392] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.799014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.825012] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.883013] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.899000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.928014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.960013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.011016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.032456] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.034265] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.036336] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.040995] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.042234] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.047099] iommu: Default domain type: Passthrough [ 1.048398] SCSI subsystem initialized [ 1.051143] ACPI: bus type USB registered [ 1.052065] usbcore: registered new interface driver usbfs [ 1.053041] usbcore: registered new interface driver hub [ 1.055079] usbcore: registered new device driver usb [ 1.057147] pps_core: LinuxPPS API ver. 1 registered [ 1.059010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.062124] PTP clock support registered [ 1.064126] EDAC MC: Ver: 3.0.0 [ 1.066108] PCI: Using ACPI for IRQ routing [ 1.068796] NetLabel: Initializing [ 1.070018] NetLabel: domain hash size = 128 [ 1.072016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.075076] NetLabel: unlabeled traffic allowed by default [ 1.078268] vgaarb: loaded [ 1.080341] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.081009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.087000] clocksource: Switched to clocksource kvm-clock [ 1.188982] VFS: Disk quotas dquot_6.6.0 [ 1.191101] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.195014] *** VALIDATE ramfs *** [ 1.196772] *** VALIDATE hugetlbfs *** [ 1.199398] pnp: PnP ACPI init [ 1.202325] pnp: PnP ACPI: found 6 devices [ 1.245606] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.250848] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.253945] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.256009] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.257604] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.259362] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.261310] NET: Registered protocol family 2 [ 1.263074] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.266902] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.269626] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.274067] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.276810] TCP: Hash tables configured (established 65536 bind 65536) [ 1.279108] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.281610] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.283751] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.285908] NET: Registered protocol family 1 [ 1.290047] RPC: Registered named UNIX socket transport module. [ 1.291502] RPC: Registered udp transport module. [ 1.292613] RPC: Registered tcp transport module. [ 1.293839] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.295502] NET: Registered protocol family 44 [ 1.297852] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.300891] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.303840] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.307051] PCI: CLS 0 bytes, default 64 [ 1.309440] Unpacking initramfs... [ 2.633282] debug: unmapping init [mem 0xffff89ae3cc54000-0xffff89ae3ffbffff] [ 2.637324] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.639182] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.642068] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 3.138152] Initialise system trusted keyrings [ 3.143205] Key type blacklist registered [ 3.145097] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.155744] zbud: loaded [ 3.159688] *** VALIDATE nfs *** [ 3.163075] *** VALIDATE nfs4 *** [ 3.166814] pstore: using deflate compression [ 3.170906] Platform Keyring initialized [ 3.303789] NET: Registered protocol family 38 [ 3.306116] Key type asymmetric registered [ 3.308382] Asymmetric key parser 'x509' registered [ 3.311015] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.315668] io scheduler mq-deadline registered [ 3.316968] io scheduler kyber registered [ 3.318130] io scheduler bfq registered [ 3.320306] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.322382] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.324317] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.326226] ACPI: Power Button [PWRF] [ 3.330445] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.335838] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.347353] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.358098] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.385090] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.415081] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.445235] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.450952] Non-volatile memory driver v1.3 [ 3.452250] Linux agpgart interface v0.103 [ 3.476513] virtio_blk virtio1: [vda] 145872 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.478827] vda: detected capacity change from 0 to 74686464 [ 3.505377] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.508375] vdb: detected capacity change from 0 to 1073741824 [ 3.520422] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.523391] vdc: detected capacity change from 0 to 2621440000 [ 3.536490] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.539586] vdd: detected capacity change from 0 to 2621440000 [ 3.552521] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.555708] vde: detected capacity change from 0 to 4294967296 [ 3.570090] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.572566] vdf: detected capacity change from 0 to 4294967296 [ 3.578380] libphy: Fixed MDIO Bus: probed [ 3.588285] usbcore: registered new interface driver usbserial_generic [ 3.590365] usbserial: USB Serial support registered for generic [ 3.591840] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.595304] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.596498] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.598927] mousedev: PS/2 mouse device common for all mice [ 3.601256] rtc_cmos 00:05: RTC can wake from S4 [ 3.603933] rtc_cmos 00:05: registered as rtc0 [ 3.605248] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.607873] intel_pstate: CPU model not supported [ 3.610549] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.614378] hid: raw HID events driver (C) Jiri Kosina [ 3.614788] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.617976] usbcore: registered new interface driver usbhid [ 3.621835] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.623302] usbhid: USB HID core driver [ 3.627665] drop_monitor: Initializing network drop monitor service [ 3.631059] Initializing XFRM netlink socket [ 3.633819] NET: Registered protocol family 10 [ 3.637483] Segment Routing with IPv6 [ 3.639425] NET: Registered protocol family 17 [ 3.642034] mpls_gso: MPLS GSO support [ 3.648343] RAS: Correctable Errors collector initialized. [ 3.649888] AVX version of gcm_enc/dec engaged. [ 3.651053] AES CTR mode by8 optimization enabled [ 3.732826] sched_clock: Marking stable (3732767877, 0)->(4716304809, -983536932) [ 3.737201] registered taskstats version 1 [ 3.740092] Loading compiled-in X.509 certificates [ 3.742838] zswap: loaded using pool lzo/zbud [ 3.767862] Key type big_key registered [ 3.778997] Key type encrypted registered [ 3.780707] ima: No TPM chip found, activating TPM-bypass! [ 3.782928] ima: Allocated hash algorithm: sha1 [ 3.784657] ima: No architecture policies found [ 3.786544] evm: Initialising EVM extended attributes: [ 3.788515] evm: security.selinux [ 3.789937] evm: security.ima [ 3.791057] evm: security.capability [ 3.792418] evm: HMAC attrs: 0x1 [ 3.795319] rtc_cmos 00:05: setting system clock to 2026-08-03 19:49:34 UTC (1785786574) [ 3.801317] debug: unmapping init [mem 0xffffffffbd403000-0xffffffffbd5fffff] [ 3.804624] debug: unmapping init [mem 0xffffffffbc182000-0xffffffffbc458fff] [ 3.812094] Write protecting the kernel read-only data: 28672k [ 3.816128] debug: unmapping init [mem 0xffffffffba803000-0xffffffffba9fffff] [ 3.819215] debug: unmapping init [mem 0xffffffffbb114000-0xffffffffbb1fffff] [ 3.852619] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.861472] systemd[1]: Detected virtualization kvm. [ 3.863463] systemd[1]: Detected architecture x86-64. [ 3.865473] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.889906] systemd[1]: No hostname configured. [ 3.891739] systemd[1]: Set hostname to . [ 3.893930] random: systemd: uninitialized urandom read (16 bytes read) [ 3.895885] systemd[1]: Initializing machine ID from random generator. [ 4.022443] random: systemd: uninitialized urandom read (16 bytes read) [ 4.024631] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.028142] random: systemd: uninitialized urandom read (16 bytes read) [ 4.030390] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.034530] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.645392] device-mapper: uevent: version 1.0.3 [ 4.647419] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.360991] virtio_net virtio0 ens2: renamed from eth0 [ 5.387648] random: fast init done [ 5.390153] scsi host0: ata_piix [ 5.421246] scsi host1: ata_piix [ 5.425828] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.430871] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.094440] dracut-initqueue[578]: RTNETLINK answers: File exists [ 10.301732] random: crng init done [ 10.303271] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.838535] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.020959] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.324537] SELinux: Disabled at runtime. [ 12.389889] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.398732] systemd[1]: Detected virtualization kvm. [ 12.400630] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.938869] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.941870] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.945989] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.949453] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.952124] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.963511] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.968339] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [ 13.059516] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd Root File System. Starting udev Coldplug all Devices... Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting Huge Pages File System... [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.410137] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.751637] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.822967] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.972034] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.985292] EDAC sbridge: Ver: 1.1.2 [ 15.804812] Key type dns_resolver registered [ 16.090838] NFS: Registering the id_resolver key type [ 16.092908] Key type id_resolver registered [ 16.094789] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg431-server login: [ 42.173824] libcfs: loading out-of-tree module taints kernel. [ 42.194429] Key type ._llcrypt registered [ 42.196090] Key type .llcrypt registered [ 42.250494] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 58.216348] hrtimer: interrupt took 3680783 ns [ 62.480696] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 64.996120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 65.032526] alg: No test for adler32 (adler32-zlib) [ 66.877585] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 68.263145] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 70.207436] Key type lgssc registered [ 73.230098] Lustre: Echo OBD driver; http://www.lustre.org/ [ 99.474262] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 153.584163] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 169.279195] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 169.335337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 170.916160] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 170.968820] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 171.113503] Lustre: lustre-MDT0000: new disk, initializing [ 171.229620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 171.263303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 178.431165] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 196.146727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 196.280027] Lustre: 6508:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 196.325769] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 196.331069] Lustre: Skipped 1 previous similar message [ 196.456330] Lustre: lustre-MDT0001: new disk, initializing [ 196.566114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 196.615672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 196.632197] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 201.679229] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 207.917931] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 220.734940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 221.029575] Lustre: lustre-OST0000: new disk, initializing [ 221.035864] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 221.047662] Lustre: 8446:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 221.160604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 229.660438] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 229.965811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 229.977966] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 230.106265] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 249.332868] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 249.636551] Lustre: lustre-OST0001: new disk, initializing [ 249.652665] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 249.673134] Lustre: 9520:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 249.905347] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 258.554307] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 259.614812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 259.634925] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 259.790240] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 277.749479] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 287.461324] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 299.050522] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing check_logdir /tmp/testlogs/ [ 308.565930] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing yml_node [ 314.323472] Lustre: DEBUG MARKER: Client: 2.17.56.1 [ 318.437370] Lustre: DEBUG MARKER: MDS: 2.17.56.1 [ 322.682824] Lustre: DEBUG MARKER: OSS: 2.17.56.1 [ 324.804690] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 3 15:54:53 EDT 2026 [ 350.909566] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 352.759624] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 372.704223] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 372.707317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 372.722457] Lustre: Skipped 3 previous similar messages [ 372.748880] Lustre: Skipped 3 previous similar messages [ 377.864055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 377.881331] Lustre: Skipped 3 previous similar messages [ 380.555502] Lustre: server umount lustre-MDT0000 complete [ 387.419873] LustreError: 7922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785786958 with bad export cookie 2425955756712216801 [ 387.435946] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 387.459835] LustreError: 7922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 388.069940] LustreError: 6513:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 388.074550] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 388.074951] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 388.106528] LustreError: 6513:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 388.186539] Lustre: Skipped 1 previous similar message [ 392.096051] LustreError: 6515:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 392.096278] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 392.167231] LustreError: 6515:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 392.212566] Lustre: Skipped 2 previous similar messages [ 395.227236] Lustre: server umount lustre-MDT0001 complete [ 401.608217] Lustre: server umount lustre-OST0000 complete [ 408.434152] Lustre: server umount lustre-OST0001 complete [ 417.484910] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 434.452878] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 494.717519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 495.550107] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 495.858799] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 496.194824] Lustre: lustre-MDT0000: new disk, initializing [ 496.455924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 496.517042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 508.602416] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 527.059809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 527.413501] Lustre: 15099:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 527.633147] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 527.653148] Lustre: Skipped 1 previous similar message [ 527.836852] Lustre: lustre-MDT0001: new disk, initializing [ 528.090646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 528.198141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 528.225655] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 535.095256] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 548.618206] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 557.093694] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 566.610403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 567.123932] Lustre: lustre-OST0000: new disk, initializing [ 567.138555] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 567.152145] Lustre: 17070:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 567.336272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 568.669982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 568.704275] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 568.830567] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 575.861803] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 589.990444] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 597.126854] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 597.431863] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 602.396379] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 602.629326] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 609.779170] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 609.785483] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 609.792344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 609.817622] Lustre: Skipped 1 previous similar message [ 611.592237] Lustre: server umount lustre-OST0000 complete [ 620.005272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 620.029206] Lustre: Skipped 1 previous similar message [ 620.040192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 620.045941] Lustre: Skipped 1 previous similar message [ 623.373925] Lustre: server umount lustre-MDT0000 complete [ 625.125180] LustreError: 15109:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 625.151419] LustreError: 15109:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 629.355405] LustreError: 15090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785787200 with bad export cookie 2425955756712218334 [ 629.365262] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 629.387096] LustreError: 15090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 630.402851] Lustre: server umount lustre-MDT0001 complete [ 638.104684] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 16:00:06 (1785787206) [ 667.542739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 668.534748] LustreError: 19951:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 668.675752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 673.765825] LustreError: 19952:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 675.952885] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 684.011621] LustreError: 19952:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 684.030634] LustreError: 19952:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 688.325895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 695.408094] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 706.560587] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 714.257718] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 717.487060] Lustre: Failing over lustre-MDT0000 [ 718.336419] Lustre: server umount lustre-MDT0000 complete [ 718.827990] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 718.843764] Lustre: Skipped 1 previous similar message [ 718.867915] LustreError: 19946:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 718.893975] LustreError: 19946:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 735.200091] Lustre: 3638:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785787289/real 1785787289] req@ffff89ad8321a300 x1872533009913472/t0(0) o400->MGC192.168.204.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785787305 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 735.255470] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 741.340129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 744.424473] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x21aab9e84fa55464 [ 744.443067] Lustre: MGC192.168.204.131@tcp: Connection restored to 0@lo (at 0@lo) [ 744.869926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 744.878096] Lustre: Skipped 1 previous similar message [ 744.975240] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 750.051177] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 750.063122] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 750.078322] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 751.450128] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 762.651774] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 768.488261] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 768.495963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 768.507044] Lustre: Skipped 2 previous similar messages [ 768.524075] Lustre: Skipped 1 previous similar message [ 774.331407] Lustre: server umount lustre-MDT0000 complete [ 778.722030] LustreError: 19948:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 778.744261] LustreError: 19948:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 779.830582] LustreError: 19934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785787350 with bad export cookie 2425955756712219748 [ 779.836375] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 779.847921] LustreError: 19934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 780.401860] Lustre: server umount lustre-MDT0001 complete [ 788.714808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 794.958941] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 807.155238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 813.753700] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 824.034536] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 831.459417] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 835.084889] Lustre: Failing over lustre-MDT0000 [ 835.740131] Lustre: server umount lustre-MDT0000 complete [ 838.624380] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 838.624466] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 838.648463] Lustre: Skipped 1 previous similar message [ 843.747064] LustreError: 23309:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 843.770393] LustreError: 23309:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 855.008823] Lustre: 3636:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785787409/real 1785787409] req@ffff89ad83073b80 x1872533009945600/t0(0) o400->MGC192.168.204.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785787425 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 855.055136] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 858.424969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 865.251468] LustreError: 3635:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff89aec0902a00 x1872533009948288/t0(0) o250->MGC192.168.204.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 865.907066] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 870.891355] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 870.926393] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 870.934324] Lustre: Skipped 1 previous similar message [ 870.953965] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 873.296164] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 885.094622] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 891.360596] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 891.369703] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 891.390246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 891.400164] Lustre: Skipped 2 previous similar messages [ 905.183144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 905.724826] Lustre: server umount lustre-MDT0000 complete [ 910.888669] LustreError: 23290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785787481 with bad export cookie 2425955756712220686 [ 910.893421] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 910.900728] LustreError: 23290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 911.403589] Lustre: server umount lustre-MDT0001 complete [ 923.718859] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 16:04:52 (1785787492) [ 946.123471] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 961.582135] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1019.184496] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1033.839698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1034.144076] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1034.166436] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1034.219650] Lustre: lustre-MDT0000: new disk, initializing [ 1034.360952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1034.367793] Lustre: Skipped 3 previous similar messages [ 1034.380237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1040.676887] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1055.755722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1055.945863] Lustre: 29734:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1056.145240] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1061.997655] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1073.344826] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1081.020639] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1090.642478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1091.163186] Lustre: 31702:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1092.726253] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1092.789609] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 1092.898901] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1092.917216] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 1100.559854] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1111.758318] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1118.637521] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 1154.537027] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1154.551794] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1154.563636] Lustre: Skipped 1 previous similar message [ 1154.576263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1154.610257] Lustre: Skipped 7 previous similar messages [ 1165.279255] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1165.774735] Lustre: server umount lustre-OST0000 complete [ 1179.104311] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1181.949464] Lustre: server umount lustre-MDT0000 complete [ 1185.274166] LustreError: 30622:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1185.303587] LustreError: 30622:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 1189.905131] LustreError: 31713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785787760 with bad export cookie 2425955756712221085 [ 1189.910368] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1189.913662] LustreError: 31713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1191.242692] Lustre: server umount lustre-MDT0001 complete [ 1209.094321] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 1214.153244] Key type lgssc unregistered [ 1214.526748] LNet: 33687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1214.538485] LNetError: 33687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1214.558896] LNet: Removed LNI 192.168.204.131@tcp [ 1216.483223] Key type .llcrypt unregistered [ 1216.490079] Key type ._llcrypt unregistered [ 1255.231892] Key type ._llcrypt registered [ 1255.235222] Key type .llcrypt registered [ 1255.369510] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1257.400341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1257.484140] alg: No test for adler32 (adler32-zlib) [ 1258.612489] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 1259.028438] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 1260.815406] Key type lgssc registered [ 1262.809448] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1274.787885] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1274.822238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1276.343841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1281.492758] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1286.657414] LustreError: 35104:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1286.677597] LustreError: 35104:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1291.746296] LustreError: 35103:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1293.977926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1294.928267] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1302.502196] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1315.240340] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1324.221811] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1333.172508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1333.783667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1339.963552] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1340.442647] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1347.194404] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1352.703958] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1371.576139] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 16:12:20 (1785787940) [ 1374.667973] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1377.483176] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1388.488471] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 16:12:36 (1785787956) [ 1406.138471] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 16:12:53 (1785787973) [ 1408.050341] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1422.252579] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1422.262727] Lustre: Skipped 2 previous similar messages [ 1424.971843] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 16:13:13 (1785787993) [ 1427.877739] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1427.885488] Lustre: Skipped 2 previous similar messages [ 1436.760227] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 1436.769249] Lustre: Skipped 1 previous similar message [ 1455.033813] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1455.042607] Lustre: Skipped 5 previous similar messages [ 1458.909429] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 16:13:47 (1785788027) [ 1462.887243] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1467.646085] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1478.280836] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 16:14:06 (1785788046) [ 1494.222889] Lustre: Setting parameter general.jobid_name=jobid_name=jobid_name=%e.root in log params [ 1494.236452] Lustre: Skipped 7 previous similar messages [ 1496.852728] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 16:14:25 (1785788065) [ 1521.132510] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1521.156684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1526.240310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1526.249365] Lustre: Skipped 2 previous similar messages [ 1527.113502] Lustre: server umount lustre-OST0000 complete [ 1535.457195] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1535.471222] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1535.498313] Lustre: Skipped 1 previous similar message [ 1535.514867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1536.483646] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1539.716301] Lustre: server umount lustre-MDT0000 complete [ 1541.608246] LustreError: 35098:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1541.627705] LustreError: 35098:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1543.422088] LustreError: 38673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785788114 with bad export cookie 13945912372947815454 [ 1543.435892] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1543.443253] LustreError: 38673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1543.870664] Lustre: server umount lustre-MDT0001 complete [ 1562.247897] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 1565.804936] Key type lgssc unregistered [ 1566.115203] LNet: 43014:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1566.122707] LNetError: 43014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1566.140186] LNet: Removed LNI 192.168.204.131@tcp [ 1567.427434] Key type .llcrypt unregistered [ 1567.433390] Key type ._llcrypt unregistered [ 1593.528549] Key type ._llcrypt registered [ 1593.530324] Key type .llcrypt registered [ 1593.612715] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 1618.704406] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1620.246590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1620.692903] alg: No test for adler32 (adler32-zlib) [ 1621.791760] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 1622.076605] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 1623.847116] Key type lgssc registered [ 1624.981490] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1668.620217] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1668.656788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1670.066988] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1670.143337] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1670.314177] Lustre: lustre-MDT0000: new disk, initializing [ 1670.452071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1670.478202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1676.909581] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1689.717601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1689.981914] Lustre: 46173:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1690.115835] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1690.122915] Lustre: Skipped 1 previous similar message [ 1690.278291] Lustre: lustre-MDT0001: new disk, initializing [ 1690.489012] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1690.538576] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1690.550563] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1696.288816] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1708.347827] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1717.010892] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1727.905061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1728.302203] Lustre: lustre-OST0000: new disk, initializing [ 1728.309941] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1728.315573] Lustre: 48140:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1728.414632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1735.274208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1735.288245] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1735.450101] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1736.971759] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1750.496193] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1757.529040] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1757.826344] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1764.403428] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1764.630193] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1771.494575] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1771.507593] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1771.509205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1771.513337] Lustre: Skipped 1 previous similar message [ 1776.624984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1776.638636] Lustre: Skipped 2 previous similar messages [ 1781.215159] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1781.357392] Lustre: server umount lustre-OST0000 complete [ 1786.849158] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1786.865694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1786.876476] Lustre: Skipped 1 previous similar message [ 1792.325355] Lustre: server umount lustre-MDT0000 complete [ 1797.096412] LustreError: 47286:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1797.114748] LustreError: 47286:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1797.534199] LustreError: 46164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785788368 with bad export cookie 11327973349412749708 [ 1797.543625] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1797.549325] LustreError: 46164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1797.854973] Lustre: server umount lustre-MDT0001 complete [ 1808.944567] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 16:19:37 (1785788377) [ 1817.942076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1818.716845] LustreError: 50604:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1818.905635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1824.242161] LustreError: 50605:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1827.205785] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1828.326432] LustreError: 50604:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1833.459658] LustreError: 50605:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1843.703309] LustreError: 50604:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1843.753132] LustreError: 50604:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1848.064947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1849.130213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1859.601829] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1869.760423] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1877.938495] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1886.388378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1886.910219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1894.536763] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1906.545820] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1912.729488] LustreError: 53140:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1914.096656] Lustre: Modifying parameter general.timeout=1 in log params [ 1915.324064] Lustre: Modifying parameter general.timeout=2 in log params [ 1918.106495] Lustre: Modifying parameter general.timeout=4 in log params [ 1918.112211] Lustre: Skipped 1 previous similar message [ 1922.301524] Lustre: Modifying parameter general.timeout=7 in log params [ 1922.317690] Lustre: Skipped 2 previous similar messages [ 1930.739370] Lustre: Modifying parameter general.timeout=14 in log params [ 1930.743929] Lustre: Skipped 6 previous similar messages [ 1946.765958] Lustre: Modifying parameter general.timeout=27 in log params [ 1946.768458] Lustre: Skipped 12 previous similar messages [ 1979.001117] Lustre: Modifying parameter general.timeout=50 in log params [ 1979.012543] Lustre: Skipped 22 previous similar messages [ 2044.283369] Lustre: Modifying parameter general.timeout=102 in log params [ 2044.291976] Lustre: Skipped 51 previous similar messages [ 2092.448374] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 16:24:20 (1785788660) [ 2096.999560] Lustre: 60122:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 2099.875183] Lustre: 60221:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 2102.103817] Lustre: 60319:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 2106.324634] Lustre: 60529:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 2106.330917] Lustre: 60529:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 2141.350797] Lustre: 61988:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 2141.358347] Lustre: 61988:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 2141.392116] LustreError: 61988:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 2148.454065] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 16:25:16 (1785788716) [ 2168.288587] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2168.295931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2168.321630] Lustre: Skipped 3 previous similar messages [ 2168.338628] Lustre: Skipped 2 previous similar messages [ 2170.666865] Lustre: server umount lustre-MDT0000 complete [ 2173.412496] LustreError: 56828:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2173.435020] LustreError: 56828:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2176.099169] LustreError: 50584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785788746 with bad export cookie 11327973349412750590 [ 2176.108535] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2176.133732] LustreError: 50584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2176.618646] Lustre: server umount lustre-MDT0001 complete [ 2192.583710] Lustre: server umount lustre-OST0000 complete [ 2229.289238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2229.815494] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2229.897876] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2229.917670] Lustre: Skipped 50 previous similar messages [ 2230.356164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2235.788995] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2248.035586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2248.299647] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2248.349277] Lustre: 64526:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2254.797960] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2265.877239] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2274.147325] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2282.070343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2282.320747] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2282.804674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2282.811644] Lustre: Skipped 1 previous similar message [ 2290.609981] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2300.730535] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2303.221275] Lustre: 67094:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 2316.259461] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2316.262993] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2316.274427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2319.328520] Lustre: server umount lustre-OST0000 complete [ 2325.472117] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2325.489863] LustreError: Skipped 1 previous similar message [ 2325.506086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2325.542600] Lustre: Skipped 1 previous similar message [ 2325.556846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2325.559360] Lustre: Skipped 1 previous similar message [ 2331.366642] Lustre: server umount lustre-MDT0000 complete [ 2331.624769] LustreError: 64537:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2331.654751] LustreError: 64537:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2337.139047] LustreError: 64518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785788907 with bad export cookie 11327973349412754041 [ 2337.143971] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2337.147164] LustreError: 64518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2337.508979] Lustre: server umount lustre-MDT0001 complete [ 2351.690144] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 2354.861692] Key type lgssc unregistered [ 2355.220608] LNet: 68307:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2355.225156] LNetError: 68307:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2355.243445] LNet: Removed LNI 192.168.204.131@tcp [ 2356.113260] Key type .llcrypt unregistered [ 2356.114766] Key type ._llcrypt unregistered [ 2388.185302] Key type ._llcrypt registered [ 2388.189845] Key type .llcrypt registered [ 2388.315141] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 2390.227453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2390.260099] alg: No test for adler32 (adler32-zlib) [ 2391.392237] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 2391.743226] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 2393.551505] Key type lgssc registered [ 2394.820937] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2403.228173] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2403.254318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2404.834855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2410.954413] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2415.076856] LustreError: 69722:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2415.091039] LustreError: 69722:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2420.199977] LustreError: 69723:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2423.410179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2423.909247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2431.173521] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2442.876500] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2453.737851] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2463.884053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2464.548023] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2472.568443] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2482.772413] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2493.098520] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2510.816348] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2510.824442] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2510.831914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2510.873190] Lustre: Skipped 1 previous similar message [ 2513.672421] Lustre: server umount lustre-MDT0000 complete [ 2515.938318] LustreError: 69722:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2515.962275] LustreError: 69722:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2518.662080] LustreError: 72410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785789089 with bad export cookie 4192453460439659246 [ 2518.663131] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2518.671651] LustreError: 72410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2519.338716] Lustre: server umount lustre-MDT0001 complete [ 2535.374096] Lustre: server umount lustre-OST0000 complete [ 2566.829857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2567.053494] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2567.075378] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2567.479746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2572.489502] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2582.647613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2582.785538] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2582.822088] Lustre: 74601:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2583.625442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2589.743774] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2598.764598] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2605.872715] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2613.784201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2614.056814] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2614.585649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2621.574444] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2630.760774] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2633.579927] Lustre: 77174:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2635.936392] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2646.496082] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2646.502873] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2646.516586] Lustre: Skipped 1 previous similar message [ 2646.524154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2646.532535] Lustre: Skipped 2 previous similar messages [ 2650.355509] Lustre: server umount lustre-OST0000 complete [ 2657.249077] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2657.259700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2657.268845] Lustre: Skipped 2 previous similar messages [ 2657.276648] Lustre: Skipped 2 previous similar messages [ 2661.208083] Lustre: server umount lustre-MDT0000 complete [ 2662.368274] LustreError: 74612:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2662.398515] LustreError: 74612:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2665.184455] LustreError: 74591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785789235 with bad export cookie 4192453460439660093 [ 2665.187203] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2665.193579] LustreError: 74591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2665.903272] Lustre: server umount lustre-MDT0001 complete [ 2681.318341] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 2684.876658] Key type lgssc unregistered [ 2685.301211] LNet: 78396:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2685.306183] LNetError: 78396:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2685.323211] LNet: Removed LNI 192.168.204.131@tcp [ 2686.404300] Key type .llcrypt unregistered [ 2686.408184] Key type ._llcrypt unregistered [ 2717.575794] Key type ._llcrypt registered [ 2717.579505] Key type .llcrypt registered [ 2717.671959] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 2718.683370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2718.795066] alg: No test for adler32 (adler32-zlib) [ 2719.791612] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 2719.963735] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 2721.592850] Key type lgssc registered [ 2723.378785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2731.881501] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2731.921125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2733.520845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2739.481485] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2743.801616] LustreError: 79811:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2743.830334] LustreError: 79811:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2748.900904] LustreError: 79810:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2750.351911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2751.189934] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2757.175870] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2770.018444] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2778.120063] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2787.593333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2788.673666] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2797.837472] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2807.194419] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2815.277042] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2816.801125] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2818.362260] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2820.785385] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2820.787963] Lustre: Skipped 1 previous similar message [ 2825.466982] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2825.471855] Lustre: Skipped 3 previous similar messages [ 2834.171787] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2834.186758] Lustre: Skipped 6 previous similar messages [ 2850.307133] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2850.320262] Lustre: Skipped 12 previous similar messages [ 2882.620861] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2882.627067] Lustre: Skipped 29 previous similar messages [ 2945.520672] Lustre: 87927:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2946.810574] Lustre: Disabling parameter general.jobid_var= in log params [ 2946.815468] Lustre: Skipped 41 previous similar messages [ 2949.577725] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 16:38:37 (1785789517) [ 2951.770279] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2954.225611] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 16:38:42 (1785789522) [ 2959.841487] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2959.853364] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2959.872873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2962.918107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2962.920481] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2962.923909] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2965.419404] Lustre: server umount lustre-OST0000 complete [ 2971.104165] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2971.113735] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2971.136616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2971.138882] Lustre: Skipped 1 previous similar message [ 2973.152610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2973.153071] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2975.885597] Lustre: server umount lustre-MDT0000 complete [ 2978.272311] LustreError: 79810:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2978.290395] LustreError: 79810:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2980.018064] LustreError: 79791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785789550 with bad export cookie 10410219035663398597 [ 2980.033703] LustreError: 79791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2980.035581] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2980.568500] Lustre: server umount lustre-MDT0001 complete [ 2994.932263] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 2998.589340] Key type lgssc unregistered [ 2998.914487] LNet: 89175:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2998.930142] LNetError: 89175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2999.976971] LNet: Removed LNI 192.168.204.131@tcp [ 3001.128558] Key type .llcrypt unregistered [ 3001.132745] Key type ._llcrypt unregistered [ 3006.312789] Key type ._llcrypt registered [ 3006.317316] Key type .llcrypt registered [ 3006.394575] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 3011.435988] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules [ 3012.913455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3012.979917] alg: No test for adler32 (adler32-zlib) [ 3014.056641] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 3014.425143] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 3016.191444] Key type lgssc registered [ 3017.805697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3019.341078] LustreError: 90390:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 3040.898580] LustreError: 90390:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 3041.795204] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 3053.589342] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3053.607828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3055.030038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3061.211738] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3065.315119] LustreError: 91407:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3065.361082] LustreError: 91407:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3070.433199] LustreError: 91408:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3075.380141] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 16:40:43 (1785789643) [ 3075.581381] LustreError: 91407:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3080.675083] LustreError: 91408:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3082.402322] LustreError: 91439:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3082.413495] LustreError: 91439:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3082.430746] LustreError: 91439:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 28, retries 0, failed: rc = -5 [ 3083.087935] Lustre: server umount lustre-MDT0000 complete [ 3105.905189] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 3110.026864] Key type lgssc unregistered [ 3110.371625] LNet: 92948:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3110.381434] LNetError: 92948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3111.404526] LNet: Removed LNI 192.168.204.131@tcp [ 3113.061467] Key type .llcrypt unregistered [ 3113.063139] Key type ._llcrypt unregistered [ 3134.560611] Key type ._llcrypt registered [ 3134.567184] Key type .llcrypt registered [ 3134.736883] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 3136.404601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3136.492485] alg: No test for adler32 (adler32-zlib) [ 3137.612253] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 3137.914232] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 3139.608429] Key type lgssc registered [ 3141.291805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3149.966442] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3149.985415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3151.865172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3157.280611] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3162.082516] LustreError: 94167:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3162.121220] LustreError: 94167:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3167.231104] LustreError: 94168:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3170.473473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3171.652590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3179.360697] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3188.650348] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3196.574772] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3207.073023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3207.677773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3216.270982] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3225.848592] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3288.439273] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 16:44:17 (1785789857) [ 3290.364969] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 3292.848213] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 16:44:21 (1785789861) [ 3298.810637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3298.812196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3298.850900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3303.904334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3303.920812] Lustre: Skipped 4 previous similar messages [ 3307.226559] Lustre: server umount lustre-MDT0000 complete [ 3309.028605] LustreError: 94168:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3309.047561] LustreError: 94168:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3311.025508] LustreError: 94148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785789881 with bad export cookie 5490484411967301492 [ 3311.040967] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3311.585391] Lustre: server umount lustre-MDT0001 complete [ 3326.716217] Lustre: server umount lustre-OST0000 complete [ 3340.765929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3341.470686] LustreError: 98471:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3341.727724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3346.471103] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3346.914260] LustreError: 98472:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3356.671549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3357.316797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3362.175716] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3371.802122] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3378.760133] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3392.800691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3393.081127] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 3393.092535] LustreError: 98461:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 3393.126376] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 3393.135570] LustreError: 100557:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 3393.148350] LustreError: 100557:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 3393.166181] LustreError: 100557:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 3393.264457] Lustre: server umount lustre-OST0000 complete [ 3393.267629] LustreError: 100557:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3404.539670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3404.817172] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 3404.824900] LustreError: 98460:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 3404.852646] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 3404.867570] LustreError: 101024:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 3404.882879] LustreError: 101024:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 3404.889129] LustreError: 101024:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 3405.027989] Lustre: server umount lustre-OST0000 complete [ 3405.034187] LustreError: 101024:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3417.596089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3417.875376] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3417.884769] Lustre: Found index 0 for lustre-OST0000, updating log [ 3417.902154] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 3417.991382] Lustre: lustre-OST0000: new disk, initializing [ 3418.002332] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3418.021043] Lustre: 101560:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3418.332239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3425.991396] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3430.794548] Lustre: Failing over lustre-OST0000 [ 3430.907173] Lustre: server umount lustre-OST0000 complete [ 3435.922756] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3435.926383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3435.943015] Lustre: Skipped 3 previous similar messages [ 3435.957435] Lustre: Skipped 1 previous similar message [ 3441.123449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3441.133938] Lustre: Skipped 1 previous similar message [ 3446.242995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3448.032823] Lustre: 93482:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785790002/real 1785790002] req@000000006541aa60 x1872536228964864/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1785790018 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3448.069137] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3450.337058] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3450.643796] Lustre: server umount lustre-MDT0000 complete [ 3451.366253] Lustre: 93480:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785790006/real 1785790006] req@0000000078c391e5 x1872536228966912/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785790022 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3453.408380] LustreError: 98466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3453.437226] LustreError: 98466:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3454.700035] LustreError: 98453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785790025 with bad export cookie 5490484411967304236 [ 3454.704089] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3454.714260] LustreError: 98453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3455.346420] Lustre: server umount lustre-MDT0001 complete [ 3465.782700] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 16:47:14 (1785790034) [ 3485.104877] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 3497.860501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3498.450429] LustreError: 103908:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3498.654824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3504.766641] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3515.083058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3520.873704] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3524.177956] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3532.746481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3533.798442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3533.810681] Lustre: Skipped 1 previous similar message [ 3533.825605] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3540.752543] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3540.983311] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3541.048281] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3541.051289] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3550.382536] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3550.537411] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3550.715866] Lustre: lustre-OST0001: new disk, initializing [ 3550.722773] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3550.730644] Lustre: 106225:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3556.433019] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3556.440161] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3556.543989] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3558.482580] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3567.718851] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3574.287181] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3582.042152] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.31@tcp (stopping) [ 3582.048904] Lustre: Skipped 3 previous similar messages [ 3582.439464] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3582.468022] Lustre: Skipped 2 previous similar messages [ 3593.184828] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3593.437620] Lustre: server umount lustre-MDT0001 complete [ 3597.365560] LustreError: 105108:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3597.388161] LustreError: 105108:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3603.589236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3603.718227] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3603.727160] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3603.822275] Lustre: 103896:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3604.929685] LustreError: 107734:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3604.953495] LustreError: 107734:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3604.970907] LustreError: 103900:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3605.015281] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3605.025980] Lustre: Skipped 1 previous similar message [ 3605.169728] LustreError: 107748:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.131@tcp: cfg command failed: rc = -17 [ 3605.194236] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3605.194236] [ 3610.107284] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 3610.295806] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3623.025945] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 16:49:51 (1785790191) [ 3644.737954] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 3665.978765] Lustre: 109841:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3691.879172] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3697.827575] Lustre: 110978:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3819.488451] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3819.501027] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3819.528409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3819.545843] Lustre: Skipped 11 previous similar messages [ 3822.612668] Lustre: server umount lustre-MDT0000 complete [ 3825.121984] LustreError: 105108:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3825.152059] LustreError: 105108:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 3828.150312] LustreError: 103889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785790398 with bad export cookie 5490484411967305237 [ 3828.152758] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3828.160561] LustreError: 103889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3829.048469] Lustre: server umount lustre-MDT0001 complete [ 3845.407191] Lustre: 93482:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785790400/real 1785790400] req@ffff89ad8d3fc000 x1872536229249024/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785790416 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3845.445316] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3845.456533] Lustre: Skipped 3 previous similar messages [ 3846.957253] Lustre: server umount lustre-OST0000 complete [ 3848.688072] Lustre: 93479:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785790403/real 1785790403] req@ffff89aeb225c000 x1872536229249280/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785790419 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3853.034713] Lustre: server umount lustre-OST0001 complete [ 3861.041988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3875.266123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3888.294209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3902.938371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3934.232733] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 3947.132930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3947.147672] Lustre: lustre-MDT0000: reset Object Index mappings [ 3947.525794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3953.464935] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3957.734229] LustreError: 114963:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3957.773588] LustreError: 114963:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3963.633037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3963.671848] Lustre: lustre-MDT0001: reset Object Index mappings [ 3970.265520] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3973.683442] Lustre: 116111:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3981.941130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3988.865110] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:231 to 0x280000400:257) [ 3990.174945] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3992.947297] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 4006.401184] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4013.580441] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:157 to 0x2c0000401:193) [ 4013.598048] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 4022.219601] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 4032.613276] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4037.877966] Lustre: 117993:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4103.807296] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 16:57:52 (1785790672) [ 4131.298239] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4131.324513] Lustre: Skipped 1 previous similar message [ 4131.328731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4131.341197] Lustre: Skipped 4 previous similar messages [ 4135.837231] Lustre: server umount lustre-MDT0000 complete [ 4141.030985] LustreError: 114945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785790711 with bad export cookie 5490484411967352123 [ 4141.043363] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4141.057830] LustreError: 114945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4147.593877] Lustre: server umount lustre-MDT0001 complete [ 4159.164220] Lustre: server umount lustre-OST0000 complete [ 4170.679222] Lustre: server umount lustre-OST0001 complete [ 4179.892217] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 4192.774624] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 4238.432862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4238.819310] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4238.880070] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4239.025920] Lustre: lustre-MDT0000: new disk, initializing [ 4239.119761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4239.139398] Lustre: Skipped 3 previous similar messages [ 4239.153116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4246.215118] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 4255.484542] Lustre: server umount lustre-MDT0000 complete [ 4267.479026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4267.966244] Lustre: Found index 0 for lustre-MDT0000, updating log [ 4267.996224] Lustre: 122267:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4268.017443] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 4274.985481] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 4281.408832] Lustre: server umount lustre-MDT0000 complete [ 4292.549741] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 17:01:00 (1785790860) [ 4294.379448] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 4296.437621] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 17:01:05 (1785790865) [ 4330.815967] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 4343.235658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4349.029318] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 4360.145749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4360.303104] Lustre: 124355:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4360.344096] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4360.347413] Lustre: Skipped 1 previous similar message [ 4360.414950] Lustre: lustre-MDT0001: new disk, initializing [ 4360.508108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4360.519838] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4366.305850] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 4371.932790] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4382.412355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4382.961687] Lustre: lustre-OST0000: new disk, initializing [ 4382.972473] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4382.993384] Lustre: 125932:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4384.342130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4384.380264] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4384.545989] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4395.752453] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 4410.594381] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4410.959587] Lustre: lustre-OST0001: new disk, initializing [ 4410.965825] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4410.980272] Lustre: 126804:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4412.666084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4412.680400] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4412.869190] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4420.035958] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 4434.740423] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4439.151799] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4935.768115] Lustre: 133241:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4935.776453] Lustre: 133241:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4935.789167] Lustre: 133241:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4935.792987] Lustre: 133241:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4935.801737] Lustre: 133241:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4935.816553] Lustre: 133241:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 5266.667461] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 17:17:13 (1785791833) [ 5287.399555] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5287.417363] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5287.453342] Lustre: Skipped 4 previous similar messages [ 5287.468222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5287.481825] Lustre: Skipped 6 previous similar messages [ 5294.989202] Lustre: server umount lustre-MDT0000 complete [ 5298.663951] LustreError: 127630:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5298.738750] LustreError: 127630:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [ 5302.094513] LustreError: 127571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785791872 with bad export cookie 5490484411967424041 [ 5302.121501] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5303.779973] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5303.783264] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5303.824790] Lustre: Skipped 4 previous similar messages [ 5303.856228] Lustre: Skipped 9 previous similar messages [ 5309.244043] Lustre: server umount lustre-MDT0001 complete [ 5322.820408] Lustre: server umount lustre-OST0000 complete [ 5342.175452] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5342.813264] Lustre: server umount lustre-OST0001 complete [ 5355.022583] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 5367.865165] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 5419.121328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5419.674977] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5419.765843] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5419.997334] Lustre: lustre-MDT0000: new disk, initializing [ 5420.202253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5420.222478] Lustre: Skipped 5 previous similar messages [ 5420.259619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5427.385843] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 5440.798675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5440.981802] Lustre: 136316:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5441.164707] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5441.174896] Lustre: Skipped 1 previous similar message [ 5441.405634] Lustre: lustre-MDT0001: new disk, initializing [ 5441.739600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5441.764499] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5448.916299] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 5461.982859] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5471.214714] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5479.639269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5479.915720] Lustre: lustre-OST0000: new disk, initializing [ 5479.920590] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5479.930237] Lustre: 138286:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5481.263147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5481.286640] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5481.393565] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5488.328147] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 5502.606315] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5524.076615] Lustre: lustre-MDD0000: changelog on [ 5528.611276] Lustre: lustre-MDD0001: changelog on [ 8254.274023] Lustre: lustre-MDD0001: changelog off [ 8256.873570] Lustre: 143464:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 8258.294822] Lustre: lustre-MDD0000: changelog off [ 8269.806740] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8269.809707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8269.824678] Lustre: Skipped 1 previous similar message [ 8269.837141] Lustre: Skipped 3 previous similar messages [ 8273.582369] Lustre: server umount lustre-OST0000 complete [ 8279.113274] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8279.114415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8279.140624] Lustre: Skipped 1 previous similar message [ 8279.152868] Lustre: Skipped 1 previous similar message [ 8284.129183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8284.144214] Lustre: Skipped 1 previous similar message [ 8286.129410] Lustre: server umount lustre-MDT0000 complete [ 8289.249704] LustreError: 140282:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8289.265637] LustreError: 140282:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 8291.496165] LustreError: 137112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785794862 with bad export cookie 5490484411968210449 [ 8291.498554] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8291.510794] LustreError: 137112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 8292.075464] Lustre: server umount lustre-MDT0001 complete [ 8307.366435] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 8311.178982] Key type lgssc unregistered [ 8311.574812] LNet: 147568:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8311.588431] LNetError: 147568:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8311.602568] LNet: Removed LNI 192.168.204.131@tcp [ 8312.650141] Key type .llcrypt unregistered [ 8312.652746] Key type ._llcrypt unregistered [ 8338.764254] Key type ._llcrypt registered [ 8338.770653] Key type .llcrypt registered [ 8338.918024] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 8359.136793] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 8361.292212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8361.374668] alg: No test for adler32 (adler32-zlib) [ 8362.886703] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 8363.389842] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 8365.183183] Key type lgssc registered [ 8367.051890] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8418.997493] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8419.030610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8420.440511] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8420.478540] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8420.574465] Lustre: lustre-MDT0000: new disk, initializing [ 8420.646692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8420.671275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8427.547579] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8443.933621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8444.165418] Lustre: 150774:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8444.273477] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8444.281344] Lustre: Skipped 1 previous similar message [ 8444.412443] Lustre: lustre-MDT0001: new disk, initializing [ 8444.624233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8444.748774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8444.770732] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8450.730217] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8462.574253] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8470.234265] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8479.495864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8479.963149] Lustre: lustre-OST0000: new disk, initializing [ 8479.967382] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8479.975762] Lustre: 152748:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8480.107312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8488.132440] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8490.091227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8490.112890] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8490.297602] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8504.473382] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8512.282069] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_module obdecho/obdecho [ 8514.841394] LustreError: 153754:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 8514.851696] LustreError: 153754:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 8517.198597] LustreError: 153850:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 8536.547839] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8536.550156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8536.566969] Lustre: Skipped 2 previous similar messages [ 8536.578964] Lustre: Skipped 2 previous similar messages [ 8539.836881] Lustre: server umount lustre-MDT0000 complete [ 8544.521897] LustreError: 150769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785795115 with bad export cookie 9994654156522774913 [ 8544.524496] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8544.539311] LustreError: 150769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8545.073217] Lustre: server umount lustre-MDT0001 complete [ 8561.281415] Lustre: server umount lustre-OST0000 complete [ 8594.668953] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 8608.427540] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 8669.840357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8670.281052] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8670.309248] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8670.443466] Lustre: lustre-MDT0000: new disk, initializing [ 8670.655289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8670.697202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8682.501886] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8698.722068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8698.924315] Lustre: 157572:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8699.057529] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8699.063585] Lustre: Skipped 1 previous similar message [ 8699.260590] Lustre: lustre-MDT0001: new disk, initializing [ 8699.513412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8699.581783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8699.603269] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8706.068487] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8719.520657] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8728.470194] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8740.935850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8741.338205] Lustre: lustre-OST0000: new disk, initializing [ 8741.348116] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8741.356739] Lustre: 159547:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8741.527753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8747.153683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8747.174664] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8747.407504] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8751.486941] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8765.191634] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8772.190886] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8772.558861] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8779.128818] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8779.430814] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8783.331716] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8783.334084] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8783.346864] LustreError: Skipped 1 previous similar message [ 8783.360745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8783.377675] Lustre: Skipped 1 previous similar message [ 8783.408820] Lustre: Skipped 1 previous similar message [ 8788.460195] LustreError: 159732:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8788.481213] LustreError: 159732:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 8788.599779] Lustre: server umount lustre-OST0000 complete [ 8796.642996] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8796.656498] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8796.670455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8798.692763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8798.697432] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8802.216772] Lustre: server umount lustre-MDT0000 complete [ 8803.815119] LustreError: 157585:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8803.842111] LustreError: 157585:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8807.099460] LustreError: 157565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785795377 with bad export cookie 9994654156522775872 [ 8807.113901] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8807.117832] LustreError: 157565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8807.745347] Lustre: server umount lustre-MDT0001 complete [ 8813.079330] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 18:16:21 (1785795381) [ 8815.910983] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 8819.007485] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 18:16:27 (1785795387) [ 8831.169372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8831.687380] LustreError: 161961:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8831.778098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8837.092652] LustreError: 161962:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8839.398820] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8842.214244] LustreError: 161961:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8852.461069] LustreError: 161962:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8852.494328] LustreError: 161962:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8853.186648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8860.357565] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8871.991534] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8881.357401] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8891.974641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8892.485404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8892.487691] Lustre: Skipped 1 previous similar message [ 8901.063236] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8911.053296] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8920.545997] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8920.561686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8920.591666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8920.598397] Lustre: Skipped 1 previous similar message [ 8924.783909] Lustre: server umount lustre-MDT0000 complete [ 8927.286187] LustreError: 163658:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8928.843068] LustreError: 161942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785795499 with bad export cookie 9994654156522776754 [ 8928.853549] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8932.447571] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.31@tcp (stopping) [ 8932.452495] Lustre: Skipped 4 previous similar messages [ 8932.834139] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8932.849204] Lustre: Skipped 2 previous similar messages [ 8935.507532] Lustre: server umount lustre-MDT0001 complete [ 8943.361874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8964.979326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8965.317531] LustreError: 165807:0:(llog.c:1655:llog_backup()) MGC192.168.204.131@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 8965.332410] Lustre: 165807:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.131@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 8972.412736] LustreError: 165813:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8972.445235] LustreError: 165813:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 9026.399784] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 9026.409451] Lustre: 165807:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.131@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 9037.903706] LustreError: 165815:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9037.923955] LustreError: 165815:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 40 previous similar messages [ 9067.011883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9072.104225] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9073.279652] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9085.833506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9094.629608] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 9104.380431] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9105.665659] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9115.898693] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9118.237526] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9126.323730] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9128.351565] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9155.590749] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9155.597757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9155.611285] Lustre: Skipped 1 previous similar message [ 9155.632542] Lustre: Skipped 3 previous similar messages [ 9162.174192] Lustre: server umount lustre-MDT0000 complete [ 9167.093416] LustreError: 161943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785795737 with bad export cookie 9994654156522783243 [ 9167.098497] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9167.119270] LustreError: 161943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9167.594151] Lustre: server umount lustre-MDT0001 complete [ 9181.860915] Lustre: server umount lustre-OST0000 complete [ 9207.186843] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 9223.013154] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 9272.716897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9273.060729] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9273.078514] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9273.199510] Lustre: lustre-MDT0000: new disk, initializing [ 9273.289822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9273.297646] Lustre: Skipped 1 previous similar message [ 9273.348706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9278.827748] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9292.763760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9293.017249] Lustre: 170954:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9293.544707] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9298.974593] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9312.028140] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9323.269837] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9334.535543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9335.133906] Lustre: lustre-OST0000: new disk, initializing [ 9335.145081] Lustre: Skipped 1 previous similar message [ 9335.161216] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9335.176526] Lustre: Skipped 2 previous similar messages [ 9335.198189] Lustre: 172925:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9343.622704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9343.638322] Lustre: Skipped 1 previous similar message [ 9343.655600] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9343.815274] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9346.292915] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9362.394204] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9370.109454] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9370.396698] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9376.789289] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9377.061454] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9379.813677] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9379.814183] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9379.820020] LustreError: Skipped 1 previous similar message [ 9379.821248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9379.821253] Lustre: Skipped 4 previous similar messages [ 9379.857589] Lustre: Skipped 2 previous similar messages [ 9386.175355] Lustre: server umount lustre-OST0000 complete [ 9400.366912] LustreError: 170960:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9400.394468] LustreError: 170960:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 52 previous similar messages [ 9400.705740] Lustre: server umount lustre-MDT0000 complete [ 9408.910182] LustreError: 171750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785795979 with bad export cookie 9994654156522800043 [ 9408.918182] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9408.947329] LustreError: 171750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9410.008330] Lustre: server umount lustre-MDT0001 complete [ 9423.824858] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 18:26:32 (1785795992) [ 9432.205612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9438.691684] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9450.323300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9458.094699] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9470.046694] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9479.189821] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9489.006414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9497.746297] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9509.122650] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9522.321590] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 18:28:10 (1785796090) [ 9528.806797] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9528.812787] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9528.830367] Lustre: Skipped 3 previous similar messages [ 9528.830710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9528.830714] Lustre: Skipped 5 previous similar messages [ 9535.170420] Lustre: server umount lustre-OST0000 complete [ 9547.694253] Lustre: server umount lustre-MDT0000 complete [ 9552.027837] LustreError: 175321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785796122 with bad export cookie 9994654156522800925 [ 9552.028760] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9552.045268] LustreError: 175321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9552.862729] Lustre: server umount lustre-MDT0001 complete [ 9571.876446] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 9576.068255] Key type lgssc unregistered [ 9576.480200] LNet: 179096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9576.491218] LNetError: 179096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9576.519131] LNet: Removed LNI 192.168.204.131@tcp [ 9577.904773] Key type .llcrypt unregistered [ 9577.910526] Key type ._llcrypt unregistered [ 9602.143521] Key type ._llcrypt registered [ 9602.148308] Key type .llcrypt registered [ 9602.271748] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 9603.666469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9603.750602] alg: No test for adler32 (adler32-zlib) [ 9604.842706] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 9605.091600] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 9606.817164] Key type lgssc registered [ 9608.870619] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9620.460300] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9620.525456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9636.320724] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 9641.441453] LustreError: 180342:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.131@tcp: failed processing log, type 4: rc = -110 [ 9646.623525] LustreError: 180342:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 9646.654055] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 9646.672189] LustreError: MGC192.168.204.131@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 9646.693366] LustreError: 180342:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 9646.711787] LustreError: 180342:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 9646.725154] LustreError: 180342:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9646.866076] Lustre: server umount lustre-OST0000 complete [ 9646.867741] LustreError: 180342:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9658.878962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9660.340731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9666.120509] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9670.627729] LustreError: 180841:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9670.641540] LustreError: 180841:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9675.755332] LustreError: 180842:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9677.265304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9677.767381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9684.377554] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9695.526987] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9703.796742] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9712.753246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9713.032582] LustreError: 182661:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 9713.047905] Lustre: 182661:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.131@tcp: invalid local config log lustre-OST0000: rc = -22 [ 9713.308592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9721.375727] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9731.885554] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9736.164751] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9736.172743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9736.180566] Lustre: Skipped 1 previous similar message [ 9736.200216] Lustre: Skipped 2 previous similar messages [ 9741.289670] LustreError: 180842:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9741.315223] LustreError: 180842:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9741.728359] Lustre: server umount lustre-MDT0000 complete [ 9746.404838] LustreError: 181571:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9746.427485] LustreError: 181571:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9747.109991] LustreError: 180822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785796317 with bad export cookie 5632337832203939057 [ 9747.127018] LustreError: 180822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9747.127672] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9747.784948] Lustre: server umount lustre-MDT0001 complete [ 9762.831282] Lustre: server umount lustre-OST0000 complete [ 9777.568717] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 9795.278475] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 9843.831917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9844.180762] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9844.217634] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9844.342480] Lustre: lustre-MDT0000: new disk, initializing [ 9844.475750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9844.504225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9849.806307] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9862.787697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9862.984618] Lustre: 186211:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9863.115140] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9863.124966] Lustre: Skipped 1 previous similar message [ 9863.302369] Lustre: lustre-MDT0001: new disk, initializing [ 9863.528082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9863.622554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9863.643225] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9870.039655] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9883.062522] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9890.317973] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9900.048379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9900.327337] Lustre: lustre-OST0000: new disk, initializing [ 9900.330239] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9900.339400] Lustre: 188180:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9900.436804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9907.299094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9907.313477] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9907.455320] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9908.493426] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9921.864555] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9930.038153] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9930.358069] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9937.733988] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9938.098692] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9943.526512] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9943.554910] Lustre: Skipped 1 previous similar message [ 9943.572408] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9943.580207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9943.592614] Lustre: Skipped 1 previous similar message [ 9947.453275] Lustre: server umount lustre-OST0000 complete [ 9955.812089] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9955.840233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9955.901988] Lustre: Skipped 1 previous similar message [ 9955.937302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9958.884188] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9958.884789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9958.923166] Lustre: Skipped 1 previous similar message [ 9962.313303] Lustre: server umount lustre-MDT0000 complete [ 9964.002436] LustreError: 186218:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9964.034635] LustreError: 186218:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9967.248898] LustreError: 186204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785796537 with bad export cookie 5632337832203939778 [ 9967.261330] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9967.261714] LustreError: 186204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9968.131789] Lustre: server umount lustre-MDT0001 complete [ 9982.138978] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 18:35:49 (1785796549) [10007.997530] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [10012.748825] Key type lgssc unregistered [10013.182235] LNet: 191131:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10013.193801] LNetError: 191131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10013.214361] LNet: Removed LNI 192.168.204.131@tcp [10014.762840] Key type .llcrypt unregistered [10014.766924] Key type ._llcrypt unregistered [10039.322162] Key type ._llcrypt registered [10039.331926] Key type .llcrypt registered [10039.476801] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [10042.046370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10042.198637] alg: No test for adler32 (adler32-zlib) [10043.670692] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [10044.287392] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [10046.119857] Key type lgssc registered [10048.583793] Lustre: Echo OBD driver; http://www.lustre.org/ [10059.561423] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10059.589458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10061.402400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10068.667458] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10071.529523] LustreError: 192351:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10071.551264] LustreError: 192351:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10072.549473] LustreError: 192352:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10077.678257] LustreError: 192351:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10081.358727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10082.167057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10087.838846] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10096.517767] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10108.581661] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10123.898474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10124.368491] Lustre: *** cfs_fail_loc=131e, val=0*** [10124.374488] LustreError: 194222:0:(llog.c:1696:llog_backup()) MGC192.168.204.131@tcp: failed to backup log lustre-OST0000: rc = -28 [10124.382597] Lustre: 194222:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.131@tcp: can't backup local config lustre-OST0000: rc = -28 [10124.551655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10133.976261] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10144.981687] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10148.833257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10148.840826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10148.857541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10149.360686] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10149.379522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10149.385598] Lustre: Skipped 1 previous similar message [10149.397945] Lustre: Skipped 1 previous similar message [10154.505055] LustreError: 192346:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10154.532595] LustreError: 192346:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10154.700191] Lustre: server umount lustre-MDT0000 complete [10159.566226] LustreError: 192333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785796730 with bad export cookie 4648310710555120023 [10159.571752] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10159.579329] LustreError: 192333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10159.592558] LustreError: 192348:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10159.604705] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10159.645596] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10159.652275] Lustre: Skipped 1 previous similar message [10160.204610] Lustre: server umount lustre-MDT0001 complete [10166.121186] Lustre: server umount lustre-OST0000 complete [10179.540717] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [10196.173047] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [10264.320425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10264.900479] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10264.958551] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10265.022603] Lustre: lustre-MDT0000: new disk, initializing [10265.073471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10265.086181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10271.626433] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10287.479155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10287.615277] Lustre: 197766:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10287.678994] Lustre: *** cfs_fail_loc=131e, val=0*** [10287.683180] Lustre: Skipped 1 previous similar message [10287.687669] LustreError: 198550:0:(llog.c:1696:llog_backup()) MGC192.168.204.131@tcp: failed to backup log lustre-MDT0001: rc = -28 [10287.694908] LustreError: 198550:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [10287.702805] Lustre: 198550:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.131@tcp: failed to copy new config lustre-MDT0001: rc = -28 [10287.734158] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10287.736592] Lustre: Skipped 1 previous similar message [10287.857771] Lustre: lustre-MDT0001: new disk, initializing [10288.069702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10288.129092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10288.144148] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10292.528473] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10308.236186] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10322.024893] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10330.613894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10330.957268] Lustre: *** cfs_fail_loc=131e, val=0*** [10330.959030] Lustre: Skipped 1 previous similar message [10330.966250] LustreError: 199715:0:(llog.c:1696:llog_backup()) MGC192.168.204.131@tcp: failed to backup log lustre-OST0000: rc = -28 [10330.976400] LustreError: 199715:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [10330.983300] Lustre: 199715:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.131@tcp: failed to copy new config lustre-OST0000: rc = -28 [10330.992521] Lustre: 199715:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [10331.007221] Lustre: lustre-OST0000: new disk, initializing [10331.009826] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10331.013880] Lustre: 199738:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10331.081024] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10336.827874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10336.841334] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10337.017996] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10339.963166] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10352.808730] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10358.636850] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10358.908675] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10365.706707] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10365.954812] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10373.087979] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10373.098790] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10373.106508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10373.160984] Lustre: Skipped 1 previous similar message [10375.093967] Lustre: server umount lustre-OST0000 complete [10383.329979] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10383.339200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10383.352258] Lustre: Skipped 1 previous similar message [10383.367920] Lustre: Skipped 2 previous similar messages [10388.037673] Lustre: server umount lustre-MDT0000 complete [10388.456736] LustreError: 197778:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10388.491444] LustreError: 197778:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10393.929740] LustreError: 197759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785796964 with bad export cookie 4648310710555120744 [10393.942651] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10393.946162] LustreError: 197759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10394.915486] Lustre: server umount lustre-MDT0001 complete [10407.665976] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 18:42:56 (1785796976) [10432.596686] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [10447.504760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10448.120124] LustreError: 202882:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10448.144102] LustreError: 202882:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10448.210864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10454.784080] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10467.017556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10474.066720] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10478.573828] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10488.458522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10489.182379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10489.193603] Lustre: Skipped 1 previous similar message [10498.133530] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10508.966027] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10509.261328] Lustre: lustre-OST0001: new disk, initializing [10509.268292] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10509.278641] Lustre: 205193:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10514.601331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10514.614777] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10514.776166] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10516.582992] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10529.298883] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10537.133716] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10539.172945] Lustre: DEBUG MARKER: ADD OST3 [10545.522202] loop: module loaded [10545.635370] loop0: detected capacity change from 0 to 204800000 [10546.025259] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10546.053288] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10546.076581] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10553.468776] Lustre: DEBUG MARKER: STOP OST3 [10554.008815] loop0: detected capacity change from 0 to 204800000 [10561.259225] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10561.628271] Lustre: lustre-OST0003: new disk, initializing [10561.637761] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [10561.653183] Lustre: 207267:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10561.817650] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [10561.827809] Lustre: Skipped 1 previous similar message [10563.068387] Lustre: *** cfs_fail_loc=2109, val=2*** [10563.212352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [10563.228831] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [10563.282951] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [10564.724460] Lustre: Failing over lustre-OST0003 [10565.082905] Lustre: server umount lustre-OST0003 complete [10565.099856] LustreError: 207353:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [10568.177791] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10568.196445] LustreError: 204550:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10568.219961] LustreError: 204550:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10569.875922] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10579.590819] Lustre: DEBUG MARKER: START OST3 again [10588.710430] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10589.076437] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [10590.303358] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [10590.414330] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [10590.426606] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [10597.361747] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10622.511075] Lustre: Failing over lustre-OST0003 [10622.964318] Lustre: server umount lustre-OST0003 complete [10625.010872] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10625.035212] Lustre: Skipped 1 previous similar message [10635.254338] LustreError: 204372:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10635.276431] LustreError: 204372:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [10645.516705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10652.424155] Lustre: server umount lustre-MDT0000 complete [10657.623364] LustreError: 202864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785797228 with bad export cookie 4648310710555121626 [10657.626583] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10657.649569] LustreError: 202864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10660.837862] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10660.862674] Lustre: Skipped 5 previous similar messages [10662.885989] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10662.895174] Lustre: Skipped 10 previous similar messages [10664.510753] Lustre: server umount lustre-MDT0001 complete [10668.772146] Lustre: server umount lustre-OST0000 complete [10673.384988] Lustre: server umount lustre-OST0001 complete [10682.761626] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [10695.568769] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [10755.052794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10755.340110] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10755.388597] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10755.625211] Lustre: lustre-MDT0000: new disk, initializing [10755.806310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10755.808826] Lustre: Skipped 1 previous similar message [10755.857921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10762.375449] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10778.373511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10778.579981] Lustre: 212660:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10778.641354] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10778.644233] Lustre: Skipped 1 previous similar message [10778.738273] Lustre: lustre-MDT0001: new disk, initializing [10778.876051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10778.893696] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10784.382407] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10802.166187] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10810.669383] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10820.645821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10820.968663] Lustre: lustre-OST0000: new disk, initializing [10820.974316] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10820.981094] Lustre: 214630:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10822.930411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10822.943545] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10823.126277] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10829.591230] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10844.446668] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10853.981988] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10854.482740] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10865.134323] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10865.309265] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10869.233865] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10869.245060] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10869.261709] Lustre: Skipped 1 previous similar message [10869.268510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10882.015851] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10882.418371] Lustre: server umount lustre-OST0000 complete [10891.763951] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10891.780565] LustreError: Skipped 1 previous similar message [10897.132126] Lustre: server umount lustre-MDT0000 complete [10899.938530] LustreError: 212671:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10899.968936] LustreError: 212671:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [10903.460876] LustreError: 212653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785797474 with bad export cookie 4648310710555123922 [10903.506750] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10903.969072] Lustre: server umount lustre-MDT0001 complete [10909.467778] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 18:51:17 (1785797477) [10930.994742] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [10945.155962] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [11001.845444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11002.257608] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11002.307419] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11002.452139] Lustre: lustre-MDT0000: new disk, initializing [11002.586534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11009.039981] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11024.408923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11024.553993] Lustre: 219250:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11025.158394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11025.171543] Lustre: Skipped 3 previous similar messages [11025.311741] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11033.667734] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11045.493092] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11051.956889] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11059.908095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11060.209086] Lustre: 221218:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11061.630298] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11061.851073] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11068.283700] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11080.059867] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11085.950393] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11086.294992] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11093.014024] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11093.379264] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11097.588266] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11097.593987] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11097.602273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11097.602281] Lustre: Skipped 8 previous similar messages [11097.643244] Lustre: Skipped 4 previous similar messages [11102.653559] Lustre: server umount lustre-OST0000 complete [11112.421526] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11112.426627] LustreError: Skipped 1 previous similar message [11114.687298] Lustre: server umount lustre-MDT0000 complete [11119.009918] LustreError: 219241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785797689 with bad export cookie 4648310710555124804 [11119.015488] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11119.022848] LustreError: 219241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [11119.947399] Lustre: server umount lustre-MDT0001 complete [11131.566266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11139.151864] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11150.873376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11159.721505] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11173.743194] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11182.149498] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11192.323593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11205.814707] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11220.989655] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11234.286913] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11234.288112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11234.312689] Lustre: Skipped 4 previous similar messages [11240.356706] Lustre: server umount lustre-OST0000 complete [11259.361448] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11259.683673] Lustre: server umount lustre-MDT0000 complete [11259.884540] LustreError: 224679:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11259.908378] LustreError: 224679:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [11265.356306] LustreError: 223420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785797836 with bad export cookie 4648310710555125686 [11265.379632] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11265.380593] LustreError: 223420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11266.257831] Lustre: server umount lustre-MDT0001 complete [11367.209780] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [11372.442779] Key type lgssc unregistered [11373.051938] LNet: 227107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11373.062273] LNetError: 227107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11373.116843] LNet: Removed LNI 192.168.204.131@tcp [11374.893189] Key type .llcrypt unregistered [11374.899795] Key type ._llcrypt unregistered [11410.797861] Key type ._llcrypt registered [11410.802959] Key type .llcrypt registered [11410.988446] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [11430.203215] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [11432.045219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11432.098488] alg: No test for adler32 (adler32-zlib) [11433.361740] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [11433.678689] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [11435.391169] Key type lgssc registered [11436.873673] Lustre: Echo OBD driver; http://www.lustre.org/ [11483.856213] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11483.901278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11485.168673] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11485.262795] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11485.399614] Lustre: lustre-MDT0000: new disk, initializing [11485.495409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11485.550851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11492.250549] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11507.973627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11508.149881] Lustre: 230475:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11508.310913] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11508.320559] Lustre: Skipped 1 previous similar message [11508.493419] Lustre: lustre-MDT0001: new disk, initializing [11508.608773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11508.683503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11508.693889] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11515.175442] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11527.304787] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11536.349638] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11545.583761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11546.066311] Lustre: lustre-OST0000: new disk, initializing [11546.073207] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11546.082465] Lustre: 232459:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11546.255510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11554.476156] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11554.946126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11554.958979] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11555.130453] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11569.607327] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11575.368312] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11575.640343] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11582.311475] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11582.583296] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11586.040959] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11586.045117] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11586.073859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11586.085297] LustreError: Skipped 1 previous similar message [11591.146705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11591.167432] Lustre: Skipped 1 previous similar message [11592.155447] Lustre: server umount lustre-OST0000 complete [11600.864330] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11600.876687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11600.902665] Lustre: Skipped 1 previous similar message [11600.915872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11600.918544] Lustre: Skipped 1 previous similar message [11605.228932] Lustre: server umount lustre-MDT0000 complete [11611.265433] LustreError: 230468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785798181 with bad export cookie 16786295564334746718 [11611.284059] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11611.290802] LustreError: 230468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11612.563618] Lustre: server umount lustre-MDT0001 complete [11647.382675] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [11663.002243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11663.745455] LustreError: 235420:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11663.767517] LustreError: 235420:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11663.852462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11668.965973] LustreError: 235421:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11669.883044] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11674.089225] LustreError: 235420:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11679.209603] LustreError: 235421:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11680.956298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11681.465584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11687.272355] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11691.218417] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11699.886782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11700.456743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11709.478361] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11723.679820] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11724.396418] Lustre: lustre-OST0001: new disk, initializing [11724.411856] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11724.444394] Lustre: 237728:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11724.755197] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11733.178389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11733.193570] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11733.441972] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11736.353693] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11753.278603] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11766.303589] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11788.245632] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 19:05:56 (1785798356) [11793.890990] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11793.907138] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11793.932605] Lustre: Skipped 1 previous similar message [11793.946292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11793.957817] Lustre: Skipped 2 previous similar messages [11799.456454] Lustre: server umount lustre-MDT0000 complete [11800.035606] LustreError: 235420:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11800.079282] LustreError: 235420:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11803.697458] LustreError: 235402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785798374 with bad export cookie 16786295564334747600 [11803.703755] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11803.711890] LustreError: 235402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11804.372515] Lustre: server umount lustre-MDT0001 complete [11819.424316] Lustre: server umount lustre-OST0000 complete [11821.092215] Lustre: 228701:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785798375/real 1785798375] req@ffff89aeba133100 x1872544927975552/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785798391 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11821.150516] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11821.165258] Lustre: Skipped 3 previous similar messages [11823.583197] Lustre: 228699:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785798378/real 1785798378] req@ffff89aeba130e00 x1872544927975808/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785798394 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11825.197171] Lustre: server umount lustre-OST0001 complete [11834.628719] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [11853.555966] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [11905.801426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11906.286303] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11906.357421] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11906.413663] Lustre: lustre-MDT0000: new disk, initializing [11906.468706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11906.486247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11913.755901] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11928.509726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11928.751030] Lustre: 241720:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11928.857888] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11928.869452] Lustre: Skipped 1 previous similar message [11929.046157] Lustre: lustre-MDT0001: new disk, initializing [11929.245059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11929.262748] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11934.339698] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11944.893361] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11951.226593] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11960.980446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11961.234378] Lustre: lustre-OST0000: new disk, initializing [11961.237908] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11961.245603] Lustre: 243689:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11962.939971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11962.963823] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11963.191128] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11972.750426] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11985.301751] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11990.447257] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11990.802888] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11996.307792] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11996.625281] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12004.326464] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12004.327650] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12004.370523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12004.380486] Lustre: Skipped 4 previous similar messages [12006.088775] Lustre: server umount lustre-OST0000 complete [12014.564346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12014.567569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12014.608409] Lustre: Skipped 2 previous similar messages [12014.639286] Lustre: Skipped 2 previous similar messages [12027.359676] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12027.722616] Lustre: server umount lustre-MDT0000 complete [12029.925338] LustreError: 241732:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12029.942877] LustreError: 241732:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12033.242296] LustreError: 242515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785798603 with bad export cookie 16786295564334749735 [12033.252993] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12033.258938] LustreError: 242515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12033.848775] Lustre: server umount lustre-MDT0001 complete [12044.996380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12045.931487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12045.947394] Lustre: Skipped 2 previous similar messages [12050.920131] LustreError: 245908:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12050.956382] LustreError: 245908:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12053.562259] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12068.862714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12076.996641] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12086.784768] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12098.265810] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12113.110640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12122.413954] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12134.806832] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12283.360642] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12283.375804] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12283.398422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12283.402612] Lustre: Skipped 4 previous similar messages [12287.423507] Lustre: server umount lustre-OST0000 complete [12295.137348] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12295.143258] LustreError: Skipped 1 previous similar message [12299.335694] Lustre: server umount lustre-MDT0000 complete [12300.769286] LustreError: 245904:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12300.803452] LustreError: 245904:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12304.803562] LustreError: 245889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785798875 with bad export cookie 16786295564334750617 [12304.815166] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12305.646690] Lustre: server umount lustre-MDT0001 complete [12408.648209] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [12413.303737] Key type lgssc unregistered [12413.876637] LNet: 249520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12413.893500] LNetError: 249520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12413.911958] LNet: Removed LNI 192.168.204.131@tcp [12415.009132] Key type .llcrypt unregistered [12415.016279] Key type ._llcrypt unregistered [12449.021485] Key type ._llcrypt registered [12449.026499] Key type .llcrypt registered [12449.237847] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [12474.223489] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [12476.046989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12476.106089] alg: No test for adler32 (adler32-zlib) [12477.741670] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [12478.206329] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [12480.047946] Key type lgssc registered [12482.649513] Lustre: Echo OBD driver; http://www.lustre.org/ [12540.919926] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [12554.142862] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12554.175624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12555.634714] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12555.722522] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12555.864709] Lustre: lustre-MDT0000: new disk, initializing [12556.083947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12556.120570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12563.198464] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12579.354297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12579.643863] Lustre: 253602:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12579.788456] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12579.799475] Lustre: Skipped 1 previous similar message [12579.979443] Lustre: lustre-MDT0001: new disk, initializing [12580.128925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12580.190709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12580.208598] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12586.776207] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12593.204854] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12603.156812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12603.620989] Lustre: lustre-OST0000: new disk, initializing [12603.625745] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12603.636868] Lustre: 255237:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12603.789792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12607.180738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12607.204110] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12607.443911] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12612.018238] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12623.915498] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12624.094158] Lustre: lustre-OST0001: new disk, initializing [12624.097326] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12624.103332] Lustre: 256109:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12624.218869] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12630.629121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12630.654748] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12630.783923] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12633.102852] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12651.268991] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12660.520610] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [12682.216112] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12682.222393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12682.227746] Lustre: Skipped 1 previous similar message [12682.246257] Lustre: Skipped 3 previous similar messages [12684.762240] Lustre: server umount lustre-MDT0000 complete [12690.860432] LustreError: 254409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785799261 with bad export cookie 17826250723163412781 [12690.865345] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12690.878403] LustreError: 254409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12691.626836] Lustre: server umount lustre-MDT0001 complete [12707.288664] Lustre: server umount lustre-OST0000 complete [12708.319081] Lustre: 251103:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785799263/real 1785799263] req@ffff89ae8fa76a00 x1872546022644480/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785799279 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12708.352222] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12708.376477] Lustre: Skipped 2 previous similar messages [12711.903412] Lustre: 251105:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785799266/real 1785799266] req@ffff89aeaad3f100 x1872546022644736/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785799282 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12712.327530] Lustre: server umount lustre-OST0001 complete [12732.578835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [12745.816440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [12758.669721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [12776.720045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [12808.771353] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [12823.695632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12823.739816] Lustre: lustre-MDT0000: reset Object Index mappings [12824.599204] LustreError: 261128:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12824.629223] LustreError: 261128:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [12824.754959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12830.187130] LustreError: 261129:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12831.766986] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12835.297806] LustreError: 261128:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12840.426110] LustreError: 261129:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12842.695564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12842.724309] Lustre: lustre-MDT0001: reset Object Index mappings [12843.655609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12849.985679] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12853.448478] Lustre: 262271:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [12863.604912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12864.232913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12868.393183] LustreError: 262625:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12870.484508] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [12875.190726] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12880.880986] LustreError: 262625:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12880.913191] LustreError: 262625:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [12886.289487] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12892.158730] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [12895.646823] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12906.054721] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12913.670472] Lustre: 264142:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [12922.850739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12922.853823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12922.876746] Lustre: Skipped 3 previous similar messages [12922.882255] Lustre: Skipped 2 previous similar messages [12927.646273] Lustre: server umount lustre-MDT0000 complete [12927.977993] LustreError: 262850:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12928.003537] LustreError: 262850:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [12932.570491] LustreError: 261108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785799503 with bad export cookie 17826250723163419571 [12932.582738] LustreError: 261108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12932.584603] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12933.110646] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12933.629729] Lustre: server umount lustre-MDT0001 complete [12940.067737] Lustre: server umount lustre-OST0000 complete [12945.577699] Lustre: server umount lustre-OST0001 complete [12971.178767] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 19:25:39 (1785799539) [12994.606749] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [13016.478402] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [13068.073360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13068.530178] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13068.593694] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13068.752150] Lustre: lustre-MDT0000: new disk, initializing [13068.897283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13068.899712] Lustre: Skipped 1 previous similar message [13068.945879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13076.815878] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [13090.717507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13090.863322] Lustre: 268425:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13090.988193] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13090.996138] Lustre: Skipped 1 previous similar message [13091.106127] Lustre: lustre-MDT0001: new disk, initializing [13091.216293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13091.225969] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13098.395532] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [13112.406774] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13121.366456] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13133.030264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13133.596577] Lustre: lustre-OST0000: new disk, initializing [13133.602076] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13133.637171] Lustre: 270399:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13133.718166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13133.723867] Lustre: Skipped 1 previous similar message [13135.113868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13135.129199] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13135.275456] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13144.475161] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [13159.040375] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13165.723684] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13165.917110] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13172.658427] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13172.937910] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13176.290355] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13176.295768] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13176.308601] Lustre: Skipped 1 previous similar message [13176.320837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13176.334748] Lustre: Skipped 1 previous similar message [13181.415767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13181.424367] Lustre: Skipped 1 previous similar message [13181.977911] Lustre: server umount lustre-OST0000 complete [13188.584095] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13188.597529] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13188.623705] Lustre: Skipped 1 previous similar message [13188.646110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13188.660042] Lustre: Skipped 1 previous similar message [13193.306584] Lustre: server umount lustre-MDT0000 complete [13196.771613] LustreError: 268437:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13196.816303] LustreError: 268437:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [13200.288580] LustreError: 268418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785799770 with bad export cookie 17826250723163421664 [13200.354556] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13201.574748] Lustre: server umount lustre-MDT0001 complete [13232.426382] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [13245.027799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13251.394762] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [13261.287736] LustreError: 273355:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13261.315690] LustreError: 273355:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [13262.782942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13263.159238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13263.167808] Lustre: Skipped 1 previous similar message [13270.817307] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [13276.752596] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13288.130871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13298.895865] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [13313.663435] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13314.024232] Lustre: lustre-OST0001: new disk, initializing [13314.037968] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [13314.054665] Lustre: 275665:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13315.428647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13315.446767] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13315.635977] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13324.852029] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [13339.294596] Lustre: DEBUG MARKER: Using TIMEOUT=20 [13346.377817] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [13354.209113] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [13354.502567] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [13365.731315] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13365.743071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13365.762229] Lustre: Skipped 1 previous similar message [13365.777572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13365.784071] Lustre: Skipped 2 previous similar messages [13368.177776] Lustre: server umount lustre-MDT0000 complete [13374.495216] LustreError: 273335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785799945 with bad export cookie 17826250723163422546 [13374.497560] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13374.507771] LustreError: 273335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [13375.288913] Lustre: server umount lustre-MDT0001 complete [13392.352433] Lustre: 251103:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785799947/real 1785799947] req@ffff89ae8fa50380 x1872546022837120/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785799963 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [13393.803028] Lustre: server umount lustre-OST0000 complete [13394.399575] Lustre: 251105:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785799949/real 1785799949] req@ffff89ae87033100 x1872546022837376/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785799965 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [13397.471111] Lustre: 251104:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785799952/real 1785799952] req@ffff89ae8fa50e00 x1872546022837632/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785799968 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [13400.627545] Lustre: server umount lustre-OST0001 complete [13426.755035] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [13441.680321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13442.163687] LustreError: 278729:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13442.183911] LustreError: 278729:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13449.841640] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [13460.430830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13466.240319] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [13469.967098] Lustre: 279870:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [13479.551377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13486.453066] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [13488.457875] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [13498.643463] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13504.513390] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [13508.999887] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [13523.369444] Lustre: DEBUG MARKER: Using TIMEOUT=20 [13528.100573] Lustre: 281750:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [13534.696456] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [13534.729342] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [13547.566147] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 19:35:15 (1785800115) [13565.922626] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13565.931456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13565.943475] Lustre: Skipped 6 previous similar messages [13565.957175] Lustre: Skipped 5 previous similar messages [13572.130254] Lustre: server umount lustre-MDT0000 complete [13577.497925] LustreError: 278709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785800148 with bad export cookie 17826250723163429161 [13577.511971] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13577.514299] LustreError: 278709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13584.371467] Lustre: server umount lustre-MDT0001 complete [13603.298224] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13603.396847] Lustre: server umount lustre-OST0000 complete [13609.652651] Lustre: server umount lustre-OST0001 complete [13619.006351] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [13646.394230] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [13717.432838] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [13731.420914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13731.809401] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13731.867432] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13731.992937] Lustre: lustre-MDT0000: new disk, initializing [13732.097229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13732.103584] Lustre: Skipped 6 previous similar messages [13732.117108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13738.618128] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [13754.744811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13755.041725] Lustre: 286049:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13755.700556] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13765.711417] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [13773.068384] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13781.829560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13782.177082] Lustre: 287684:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13783.831390] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13793.021781] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [13808.632128] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13809.094693] Lustre: 288556:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13810.353061] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13810.368622] Lustre: Skipped 1 previous similar message [13810.547046] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13821.168573] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [13834.135320] Lustre: DEBUG MARKER: Using TIMEOUT=20 [13839.843091] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [13864.195170] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 19:40:32 (1785800432) [13868.003579] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13868.011504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13868.024825] Lustre: Skipped 3 previous similar messages [13868.036914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13868.039515] Lustre: Skipped 8 previous similar messages [13875.719604] Lustre: server umount lustre-MDT0000 complete [13877.234433] LustreError: 286885:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13877.277763] LustreError: 286885:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 21 previous similar messages [13881.125063] LustreError: 286853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785800451 with bad export cookie 17826250723163436805 [13881.143680] LustreError: 286853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13881.151337] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13881.918719] Lustre: server umount lustre-MDT0001 complete [13898.322850] Lustre: server umount lustre-OST0000 complete [13898.722562] Lustre: 251106:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785800453/real 1785800453] req@ffff89aeb6be2a00 x1872546023016192/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785800469 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [13903.793678] Lustre: server umount lustre-OST0001 complete [13915.614392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13923.235224] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [13936.497371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13945.069581] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [13956.921415] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13965.761849] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13976.164903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13985.809270] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [13997.433612] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14021.409177] loop1: detected capacity change from 0 to 204800000 [14021.724154] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14021.743223] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14021.764631] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14029.104905] loop1: detected capacity change from 0 to 204800000 [14036.728772] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14036.901959] LustreError: 290795:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs16966: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [14036.923149] LustreError: 290795:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs16966 [14036.931496] LustreError: 290795:0:(mgs_handler.c:517:mgs_target_reg()) fs16966-MDT0001: Error checking target index: rc = -13 [14036.947439] LustreError: fs16966-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [14036.962831] LustreError: 294529:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [14036.976050] LustreError: 294529:0:(tgt_mount.c:2129:server_put_super()) no obd fs16966-MDT0001 [14036.986903] LustreError: 294529:0:(tgt_mount.c:116:server_deregister_mount()) fs16966-MDT0001 not registered [14037.101767] Lustre: server umount fs16966-MDT0001 complete [14037.107148] LustreError: 294529:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [14054.638728] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14055.009951] Lustre: Setting parameter fs16966-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs16966-MDT0001 [14055.158824] Lustre: srv-fs16966-MDT0001: No data found on store. Initialize space. [14055.165093] Lustre: Skipped 4 previous similar messages [14055.321517] Lustre: fs16966-MDT0001: new disk, initializing [14055.328557] Lustre: Skipped 3 previous similar messages [14055.539294] Lustre: srv-fs16966-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14055.549897] Lustre: 294998:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs16966-MDT0001-mdtlov: can't create new llog: rc = -115 [14055.568289] LustreError: 294998:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs16966-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [14061.875198] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [14062.062978] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [14071.419491] Lustre: server umount fs16966-MDT0001 complete [14088.824047] loop1: detected capacity change from 0 to 204800000 [14096.210087] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14096.983786] Lustre: 296566:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs16966-MDT0001-mdtlov: can't create new llog: rc = -115 [14097.002110] LustreError: 296566:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs16966-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [14103.755962] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [14110.910866] Lustre: server umount fs16966-MDT0001 complete [14126.682334] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.31@tcp (stopping) [14126.695865] Lustre: Skipped 4 previous similar messages [14128.609164] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14128.625088] Lustre: Skipped 6 previous similar messages [14133.336984] Lustre: server umount lustre-MDT0000 complete [14145.925424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14146.050233] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14151.676650] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [14155.556184] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [14169.117642] loop1: detected capacity change from 0 to 204800000 [14179.441145] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14180.006237] Lustre: 299016:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs16966-MDT0001-mdtlov: can't create new llog: rc = -115 [14180.024240] LustreError: 299016:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs16966-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [14186.279441] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [14193.911868] Lustre: server umount fs16966-MDT0001 complete [14213.100421] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [14213.115494] LustreError: Skipped 1 previous similar message [14228.747201] Lustre: server umount lustre-OST0000 complete [14233.568472] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14233.582107] LustreError: Skipped 1 previous similar message [14240.321223] Lustre: server umount lustre-MDT0000 complete [14245.041469] LustreError: 290789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785800815 with bad export cookie 17826250723163439493 [14245.060690] LustreError: 290789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14245.082294] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14245.674511] Lustre: server umount lustre-MDT0001 complete [14260.920766] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [14264.901116] Key type lgssc unregistered [14265.338372] LNet: 301365:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14265.346991] LNetError: 301365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14265.368751] LNet: Removed LNI 192.168.204.131@tcp [14266.568132] Key type .llcrypt unregistered [14266.569479] Key type ._llcrypt unregistered [14291.045928] Key type ._llcrypt registered [14291.050333] Key type .llcrypt registered [14291.212687] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [14293.227912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14293.265184] alg: No test for adler32 (adler32-zlib) [14294.698766] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [14295.051431] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [14296.751179] Key type lgssc registered [14298.726281] Lustre: Echo OBD driver; http://www.lustre.org/ [14313.746841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14313.829314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14315.821306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14322.683892] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [14326.243960] LustreError: 302638:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14326.285374] LustreError: 302638:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14331.361590] LustreError: 302639:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14335.297812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14335.905222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14341.255834] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [14354.938160] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14364.110995] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14372.956841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14373.838364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14380.987974] LustreError: 304471:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14381.526972] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [14386.155836] LustreError: 304471:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14386.195213] LustreError: 304471:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14391.130837] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14391.287940] LustreError: 304472:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14391.322698] LustreError: 304472:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14401.523018] LustreError: 304473:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14401.557057] LustreError: 304473:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [14414.871908] loop1: detected capacity change from 0 to 204800000 [14415.162112] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14415.182833] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14415.208327] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14418.515921] LustreError: 304472:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [14418.545744] LustreError: 304472:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [14421.755909] loop1: detected capacity change from 0 to 204800000 [14428.618937] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14428.807359] Lustre: fs31547-OST0000: new disk, initializing [14428.815598] Lustre: srv-fs31547-OST0000: No data found on store. Initialize space. [14428.828651] Lustre: 306377:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14428.916239] Lustre: fs31547-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14437.689499] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [14449.642703] Lustre: server umount fs31547-OST0000 complete [14450.676153] LustreError: 304473:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14450.717677] LustreError: 304473:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [14460.914351] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [14469.810769] loop1: detected capacity change from 0 to 204800000 [14480.171719] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14480.389781] Lustre: fs31547-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14492.366098] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [14504.685191] loop2: detected capacity change from 0 to 204800000 [14505.048606] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14505.067632] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14505.103738] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14513.457869] loop2: detected capacity change from 0 to 204800000 [14515.774659] LustreError: 304795:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [14515.808710] LustreError: 304795:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 36 previous similar messages [14522.884215] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14523.167605] LustreError: fs31547-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [14523.201686] LustreError: fs31547-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [14523.221783] LustreError: 309427:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [14523.246501] LustreError: 309427:0:(tgt_mount.c:2129:server_put_super()) no obd fs31547-OST0001 [14523.263709] LustreError: 309427:0:(tgt_mount.c:116:server_deregister_mount()) fs31547-OST0001 not registered [14523.354227] Lustre: server umount fs31547-OST0001 complete [14523.364196] LustreError: 309427:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [14547.747929] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14548.200629] Lustre: fs31547-OST0001: new disk, initializing [14548.207976] Lustre: srv-fs31547-OST0001: No data found on store. Initialize space. [14548.221922] Lustre: 309979:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14548.332381] Lustre: fs31547-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14558.610211] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [14574.125298] Lustre: server umount fs31547-OST0000 complete [14591.820862] Lustre: server umount fs31547-OST0001 complete [14610.698981] loop1: detected capacity change from 0 to 204800000 [14614.508862] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [14614.521930] LustreError: Skipped 1 previous similar message [14616.902801] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14617.523853] Lustre: fs31547-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14626.924817] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [14637.146295] loop2: detected capacity change from 0 to 204800000 [14643.782180] LustreError: 304471:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [14643.818730] LustreError: 304471:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 74 previous similar messages [14645.938388] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14658.911564] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [14678.079957] Lustre: server umount fs31547-OST0000 complete [14690.787896] Lustre: server umount fs31547-OST0001 complete [14711.789043] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14711.836093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14711.850709] Lustre: Skipped 1 previous similar message [14716.921689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14716.940752] Lustre: Skipped 1 previous similar message [14722.019605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14722.041868] Lustre: Skipped 1 previous similar message [14725.087173] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14725.956438] Lustre: server umount lustre-OST0000 complete [14732.256400] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14732.285451] Lustre: Skipped 1 previous similar message [14732.297351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14737.379202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14737.383128] Lustre: Skipped 2 previous similar messages [14738.768859] Lustre: server umount lustre-MDT0000 complete [14743.786201] LustreError: 311230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785801314 with bad export cookie 3929173170682852765 [14743.809574] LustreError: 311230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14743.809709] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14744.511506] Lustre: server umount lustre-MDT0001 complete [14762.627534] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [14768.110647] Key type lgssc unregistered [14768.605657] LNet: 315934:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14768.617673] LNetError: 315934:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14768.660364] LNet: Removed LNI 192.168.204.131@tcp [14770.873153] Key type .llcrypt unregistered [14770.884179] Key type ._llcrypt unregistered [14815.448343] Key type ._llcrypt registered [14815.453318] Key type .llcrypt registered [14815.644603] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [14837.344588] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [14839.772495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14839.806441] alg: No test for adler32 (adler32-zlib) [14840.994826] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [14841.545180] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [14843.351184] Key type lgssc registered [14845.314751] Lustre: Echo OBD driver; http://www.lustre.org/ [14896.054753] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14896.083131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14897.409380] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14897.439134] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14897.494948] Lustre: lustre-MDT0000: new disk, initializing [14897.577393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14897.610433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14905.509679] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [14922.069198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14922.242056] Lustre: 319796:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14922.348744] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14922.352405] Lustre: Skipped 1 previous similar message [14922.513125] Lustre: lustre-MDT0001: new disk, initializing [14922.744448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14922.836618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14922.853504] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14929.259158] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [14940.334993] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14947.024675] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14954.295262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14954.798708] Lustre: lustre-OST0000: new disk, initializing [14954.803104] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14954.810385] Lustre: 321765:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14954.948590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14963.062050] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [14963.304586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14963.326924] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14963.513049] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14976.767432] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14980.206520] Lustre: Mounted lustre-client [14981.466784] Lustre: Unmounted lustre-client [14999.523406] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14999.524032] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14999.538617] LustreError: Skipped 1 previous similar message [14999.543573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14999.602903] Lustre: Skipped 1 previous similar message [15000.551629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15000.562755] Lustre: Skipped 2 previous similar messages [15002.187727] Lustre: server umount lustre-OST0000 complete [15009.760796] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15009.774742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15009.850566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15010.786297] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15015.035648] Lustre: server umount lustre-MDT0000 complete [15021.025054] LustreError: 319807:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15021.067127] LustreError: 319807:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [15022.586464] LustreError: 319788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785801593 with bad export cookie 5965428056483604169 [15022.614449] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15022.650806] LustreError: 319788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15024.389630] Lustre: server umount lustre-MDT0001 complete [15035.167592] LNet: 1 local NIs in recovery (showing 1): 192.168.204.131@tcp [15044.683199] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [15048.826460] Key type lgssc unregistered [15049.224355] LNet: 323956:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15049.227884] LNetError: 323956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15111.727442] LNet: Removed LNI 192.168.204.131@tcp [15113.030862] Key type .llcrypt unregistered [15113.033369] Key type ._llcrypt unregistered [15138.144206] Key type ._llcrypt registered [15138.146593] Key type .llcrypt registered [15138.254868] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [15159.506960] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [15161.108815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15161.306403] alg: No test for adler32 (adler32-zlib) [15162.372626] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [15162.645728] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [15164.375160] Key type lgssc registered [15166.055612] Lustre: Echo OBD driver; http://www.lustre.org/ [15211.343119] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 20:02:58 (1785801778) [15224.419391] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15224.476111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15225.893209] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15225.977601] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15226.155096] Lustre: lustre-MDT0000: new disk, initializing [15226.370563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15226.410162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15232.956010] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [15246.512270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15246.707419] Lustre: 327192:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15246.817990] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15246.826000] Lustre: Skipped 1 previous similar message [15246.948123] Lustre: lustre-MDT0001: new disk, initializing [15247.087593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15247.133837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15247.142545] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15253.066752] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [15266.253530] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15274.461366] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15283.411527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15283.920833] Lustre: lustre-OST0000: new disk, initializing [15283.929514] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15283.951510] Lustre: 329164:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15284.131244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15291.010260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15291.023438] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15291.302896] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15292.059677] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [15310.367122] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15332.322532] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15332.332769] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15332.338605] LustreError: Skipped 1 previous similar message [15332.339650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15332.384237] Lustre: Skipped 1 previous similar message [15334.455323] Lustre: server umount lustre-OST0000 complete [15342.566338] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15342.573202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15342.589610] Lustre: Skipped 1 previous similar message [15342.605640] Lustre: Skipped 2 previous similar messages [15346.213329] Lustre: server umount lustre-MDT0000 complete [15350.566153] LustreError: 327183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785801921 with bad export cookie 12202938774261605072 [15350.572957] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15350.584053] LustreError: 327183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15351.537111] Lustre: server umount lustre-MDT0001 complete [15366.519841] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [15371.999939] Key type lgssc unregistered [15372.646059] LNet: 331073:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15372.664177] LNetError: 331073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15372.704045] LNet: Removed LNI 192.168.204.131@tcp [15374.572963] Key type .llcrypt unregistered [15374.574405] Key type ._llcrypt unregistered [15407.897136] Key type ._llcrypt registered [15407.899164] Key type .llcrypt registered [15408.023677] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [15409.762571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15409.820043] alg: No test for adler32 (adler32-zlib) [15410.839471] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [15411.047143] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [15412.697533] Key type lgssc registered [15414.837761] Lustre: Echo OBD driver; http://www.lustre.org/ [15425.072625] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15425.113905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15426.834711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15433.147855] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [15437.285748] LustreError: 332351:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15437.316218] LustreError: 332351:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15442.429273] LustreError: 332350:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15446.027087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15446.870762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15456.158178] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [15468.581163] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15477.262982] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15486.100906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15486.391355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15493.690479] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [15502.956872] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15516.382568] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15519.117135] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [15525.486376] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.31@tcp (stopping) [15528.449470] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15528.454899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15528.469678] Lustre: Skipped 1 previous similar message [15528.477448] Lustre: Skipped 2 previous similar messages [15530.557538] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.31@tcp (stopping) [15532.307521] Lustre: server umount lustre-MDT0000 complete [15533.545959] LustreError: 333079:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15533.592212] LustreError: 333079:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15535.668214] LustreError: 332347:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [15538.014586] LustreError: 333086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785802108 with bad export cookie 12457205923654239957 [15538.031242] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15538.037429] LustreError: 333086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15538.669364] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15538.687349] Lustre: Skipped 1 previous similar message [15538.696388] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [15540.785256] LustreError: 332347:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [15540.814219] LustreError: 332347:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15543.782430] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [15543.806088] Lustre: Skipped 1 previous similar message [15544.683817] Lustre: server umount lustre-MDT0001 complete [15550.404365] Lustre: server umount lustre-OST0000 complete [15561.779891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15570.573664] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [15575.775963] Lustre: 336537:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.131@tcp [15579.836634] Lustre: server umount MGS complete [15593.385309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15594.177976] LustreError: 337110:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15594.206991] LustreError: 337110:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15594.304795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15599.949273] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [15611.444050] LustreError: 337105:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [15611.465942] LustreError: 337105:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [15612.038134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15612.507647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15617.685168] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [15630.336417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15630.825313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15639.693839] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [15649.404103] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15651.537484] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [15661.274606] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15663.690742] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [15673.826321] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15673.856848] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15673.884283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15674.858283] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15678.411764] Lustre: server umount lustre-OST0000 complete [15685.092471] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15685.118467] Lustre: Skipped 1 previous similar message [15690.211042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15690.222379] Lustre: Skipped 5 previous similar messages [15690.808859] Lustre: server umount lustre-MDT0000 complete [15695.330419] LustreError: 338081:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15695.366303] LustreError: 338081:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15696.438972] LustreError: 337090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785802267 with bad export cookie 12457205923654240797 [15696.457863] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15697.076752] Lustre: server umount lustre-MDT0001 complete [15713.799221] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [15718.948501] Key type lgssc unregistered [15719.480160] LNet: 340552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15719.495355] LNetError: 340552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15719.522756] LNet: Removed LNI 192.168.204.131@tcp [15720.757199] Key type .llcrypt unregistered [15720.763290] Key type ._llcrypt unregistered [15751.297919] Key type ._llcrypt registered [15751.304237] Key type .llcrypt registered [15751.511230] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [15773.781631] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [15774.675714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15774.724225] alg: No test for adler32 (adler32-zlib) [15776.170564] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [15776.450471] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [15778.249139] Key type lgssc registered [15780.026416] Lustre: Echo OBD driver; http://www.lustre.org/ [15838.911091] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15838.944386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15840.284506] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15840.329585] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15840.423747] Lustre: lustre-MDT0000: new disk, initializing [15840.605548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15840.658524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15849.402378] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [15863.777891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15863.995972] Lustre: 343719:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15864.154930] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15864.169398] Lustre: Skipped 1 previous similar message [15864.370937] Lustre: lustre-MDT0001: new disk, initializing [15864.568248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15864.671840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15864.691954] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15870.331906] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [15885.643668] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15894.540974] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15903.757707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15904.092751] Lustre: lustre-OST0000: new disk, initializing [15904.097790] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15904.106360] Lustre: 345694:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15904.244704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15912.270532] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [15913.038705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15913.049151] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15913.148554] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15923.833862] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15930.959851] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15931.282916] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15937.164451] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15937.465788] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15944.161184] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15944.177891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15944.185655] Lustre: Skipped 1 previous similar message [15945.867275] Lustre: server umount lustre-OST0000 complete [15951.842359] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15951.852185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15951.878190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15951.882839] Lustre: Skipped 1 previous similar message [15954.402457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15954.408681] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15957.391266] Lustre: server umount lustre-MDT0000 complete [15962.742487] LustreError: 343711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785802533 with bad export cookie 7669423458081115403 [15962.747816] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15962.758436] LustreError: 343711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15963.510469] Lustre: server umount lustre-MDT0001 complete [15975.442701] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 20:15:43 (1785802543) [16001.011324] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [16012.709208] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [16058.873883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16059.141983] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16059.167847] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16059.234809] Lustre: lustre-MDT0000: new disk, initializing [16059.351527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16059.387399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16064.737227] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [16076.936843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16077.168514] Lustre: 350458:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16077.270207] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16077.277251] Lustre: Skipped 1 previous similar message [16077.412557] Lustre: lustre-MDT0001: new disk, initializing [16077.589741] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16077.670542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16077.686248] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16083.480237] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [16094.497298] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16102.275902] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16114.538726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16114.786534] Lustre: lustre-OST0000: new disk, initializing [16114.789481] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16114.793793] Lustre: 352425:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16114.879111] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16121.643546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16121.692607] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16121.985273] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16129.006995] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [16140.780523] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16193.507278] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16193.513791] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16193.545318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16193.578720] Lustre: Skipped 2 previous similar messages [16196.345127] Lustre: server umount lustre-OST0000 complete [16203.746070] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16203.766533] Lustre: Skipped 1 previous similar message [16203.774102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16208.872635] LustreError: 350469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16208.902941] LustreError: 350469:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [16209.372441] Lustre: server umount lustre-MDT0000 complete [16214.004372] LustreError: 350465:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16214.022061] LustreError: 350465:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16214.141183] LustreError: 350450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785802784 with bad export cookie 7669423458081116285 [16214.151389] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16214.707793] Lustre: server umount lustre-MDT0001 complete [16234.789493] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [16242.198850] Key type lgssc unregistered [16242.779324] LNet: 354625:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16242.799737] LNetError: 354625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16242.819992] LNet: Removed LNI 192.168.204.131@tcp [16244.796128] Key type .llcrypt unregistered [16244.802744] Key type ._llcrypt unregistered [16378.429381] Key type ._llcrypt registered [16378.432505] Key type .llcrypt registered [16378.583833] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [16378.904646] Key type .llcrypt unregistered [16378.908438] Key type ._llcrypt unregistered [16484.019265] Key type ._llcrypt registered [16484.029946] Key type .llcrypt registered [16484.257608] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [16484.929834] Key type .llcrypt unregistered [16484.938182] Key type ._llcrypt unregistered [16528.697741] Key type ._llcrypt registered [16528.708514] Key type .llcrypt registered [16528.834117] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [16529.087492] Key type .llcrypt unregistered [16529.091223] Key type ._llcrypt unregistered [16576.241403] Key type ._llcrypt registered [16576.246121] Key type .llcrypt registered [16576.501479] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [16576.852406] Key type .llcrypt unregistered [16576.856364] Key type ._llcrypt unregistered [16698.496091] Key type ._llcrypt registered [16698.499458] Key type .llcrypt registered [16698.721900] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [16699.189219] Key type .llcrypt unregistered [16699.195807] Key type ._llcrypt unregistered [16743.043773] Key type ._llcrypt registered [16743.049932] Key type .llcrypt registered [16743.195727] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [16743.498246] Key type .llcrypt unregistered [16743.499980] Key type ._llcrypt unregistered [16787.308109] Key type ._llcrypt registered [16787.310695] Key type .llcrypt registered [16787.505834] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [16787.742311] Key type .llcrypt unregistered [16787.743880] Key type ._llcrypt unregistered [16826.618842] Key type ._llcrypt registered [16826.620039] Key type .llcrypt registered [16826.755758] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [16827.162295] Key type .llcrypt unregistered [16827.171216] Key type ._llcrypt unregistered [16870.289735] Key type ._llcrypt registered [16870.292656] Key type .llcrypt registered [16870.454642] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [16871.016277] Key type .llcrypt unregistered [16871.026331] Key type ._llcrypt unregistered [16913.238561] Key type ._llcrypt registered [16913.241934] Key type .llcrypt registered [16913.368808] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [16913.602254] Key type .llcrypt unregistered [16913.604045] Key type ._llcrypt unregistered [17002.330412] Key type ._llcrypt registered [17002.331947] Key type .llcrypt registered [17002.422964] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [17002.770469] Key type .llcrypt unregistered [17002.773958] Key type ._llcrypt unregistered [17049.365776] Key type ._llcrypt registered [17049.367622] Key type .llcrypt registered [17049.439521] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [17049.896492] Key type .llcrypt unregistered [17049.900189] Key type ._llcrypt unregistered [17090.604939] Key type ._llcrypt registered [17090.615461] Key type .llcrypt registered [17090.891938] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [17091.526299] Key type .llcrypt unregistered [17091.528256] Key type ._llcrypt unregistered [17162.321766] Key type ._llcrypt registered [17162.326531] Key type .llcrypt registered [17162.472385] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [17182.686908] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [17184.251843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [17184.599648] alg: No test for adler32 (adler32-zlib) [17186.039604] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [17186.381961] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [17188.215161] Key type lgssc registered [17189.850226] Lustre: Echo OBD driver; http://www.lustre.org/ [17247.299654] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17247.319523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17248.759769] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [17248.835365] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [17249.136282] Lustre: tst250fs-MDT0000: new disk, initializing [17249.369352] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17249.434046] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [17257.508104] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [17273.063903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17273.301322] Lustre: 375069:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [17273.382127] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [17273.389263] Lustre: Skipped 1 previous similar message [17273.521898] Lustre: tst250fs-MDT0001: new disk, initializing [17273.713510] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17273.773143] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [17273.789223] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [17279.452460] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [17285.410922] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [17294.256462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17294.713751] Lustre: tst250fs-OST0000: new disk, initializing [17294.720439] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [17294.728823] Lustre: 376706:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17294.803467] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17297.216708] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [17297.230964] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [17297.396255] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [17303.357403] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [17316.332453] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17316.697348] Lustre: tst250fs-OST0001: new disk, initializing [17316.704463] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [17316.714626] Lustre: 377577:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17316.922491] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [17322.147611] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [17322.171740] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [17322.405516] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [17325.810500] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [17332.692350] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [17334.970840] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [17334.974355] Lustre: Skipped 1 previous similar message [17358.305723] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17358.310395] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [17358.336523] Lustre: Skipped 1 previous similar message [17358.350588] Lustre: Skipped 2 previous similar messages [17363.436471] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [17363.451157] Lustre: Skipped 3 previous similar messages [17364.291744] Lustre: server umount tst250fs-MDT0000 complete [17370.162756] LustreError: 375062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785803940 with bad export cookie 4123705900851856683 [17370.194362] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17370.200217] LustreError: 375062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [17371.138681] Lustre: server umount tst250fs-MDT0001 complete [17388.375366] Lustre: server umount tst250fs-OST0000 complete [17390.051699] Lustre: 373289:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785803944/real 1785803944] req@ffff89ae8971ca80 x1872550960374912/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785803960 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [17390.113131] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17390.145654] Lustre: Skipped 2 previous similar messages [17391.903765] Lustre: 373291:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785803946/real 1785803946] req@ffff89aeb16eb480 x1872550960375168/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785803962 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [17395.124936] Lustre: server umount tst250fs-OST0001 complete [17420.114561] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [17431.518991] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [17479.269169] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 20:40:48 (1785804048) [17481.923251] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [17509.073301] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [17521.415983] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [17563.726182] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 17237 sec ========= 20:42:12 (1785804132) [17566.265603] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 20:42:14 (1785804134) === [17571.082353] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 20:42:19 (1785804139) === [17632.434810] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [17636.922637] Key type lgssc unregistered [17637.263494] LNet: 386715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17637.284060] LNetError: 386715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17638.313194] LNet: Removed LNI 192.168.204.131@tcp [17639.640555] Key type .llcrypt unregistered [17639.642107] Key type ._llcrypt unregistered