[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 508595177 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003225] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.011126] LSM: Security Framework initializing [ 0.012056] Yama: becoming mindful. [ 0.013024] SELinux: Initializing. [ 0.014067] *** VALIDATE selinux *** [ 0.022829] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027596] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028124] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029120] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030123] *** VALIDATE tmpfs *** [ 0.031507] *** VALIDATE proc *** [ 0.033008] *** VALIDATE cgroup *** [ 0.034009] *** VALIDATE cgroup2 *** [ 0.035253] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036148] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038033] Spectre V2 : User space: Vulnerable [ 0.039009] Speculative Store Bypass: Vulnerable [ 0.041850] debug: unmapping init [mem 0xffffffff99c59000-0xffffffff99c60fff] [ 0.044211] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045795] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046025] ... version: 2 [ 0.047016] ... bit width: 48 [ 0.048012] ... generic registers: 4 [ 0.049013] ... value mask: 0000ffffffffffff [ 0.050014] ... max period: 00007fffffffffff [ 0.051015] ... fixed-purpose events: 3 [ 0.052012] ... event mask: 000000070000000f [ 0.053342] rcu: Hierarchical SRCU implementation. [ 0.055650] smp: Bringing up secondary CPUs ... [ 0.056779] x86: Booting SMP configuration: [ 0.057029] .... node #0, CPUs: #1 #2 #3 [ 0.061027] smp: Brought up 1 node, 4 CPUs [ 0.063019] smpboot: Max logical packages: 1 [ 0.064017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.148449] node 0 deferred pages initialised in 79ms [ 0.152205] devtmpfs: initialized [ 0.153386] x86/mm: Memory block size: 128MB [ 0.156882] gcov: version magic: 0x41383552 [ 0.160280] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.163104] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.166322] pinctrl core: initialized pinctrl subsystem [ 0.168179] [ 0.168791] ************************************************************* [ 0.171016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.173023] ** ** [ 0.176016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.178011] ** ** [ 0.180012] ** This means that this kernel is built to expose internal ** [ 0.182011] ** IOMMU data structures, which may compromise security on ** [ 0.185022] ** your system. ** [ 0.188012] ** ** [ 0.190019] ** If you see this message and you are not debugging the ** [ 0.193013] ** kernel, report this immediately to your vendor! ** [ 0.194011] ** ** [ 0.196014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.199014] ************************************************************* [ 0.201813] NET: Registered protocol family 16 [ 0.203445] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.206061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.208060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.212031] cpuidle: using governor menu [ 0.214000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.216677] PCI: Using configuration type 1 for base access [ 0.219141] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.228084] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.230041] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.233119] cryptd: max_cpu_qlen set to 1000 [ 0.236251] ACPI: Added _OSI(Module Device) [ 0.238025] ACPI: Added _OSI(Processor Device) [ 0.240010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.241012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.246180] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.251537] ACPI: Interpreter enabled [ 0.252056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.253011] ACPI: Using IOAPIC for interrupt routing [ 0.255121] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.257454] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.267793] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.270047] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.272021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.276145] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.284114] acpiphp: Slot [2] registered [ 0.285130] acpiphp: Slot [5] registered [ 0.287111] acpiphp: Slot [6] registered [ 0.288092] acpiphp: Slot [7] registered [ 0.290104] acpiphp: Slot [8] registered [ 0.291108] acpiphp: Slot [9] registered [ 0.293137] acpiphp: Slot [10] registered [ 0.294127] acpiphp: Slot [3] registered [ 0.296106] acpiphp: Slot [4] registered [ 0.298181] acpiphp: Slot [11] registered [ 0.299095] acpiphp: Slot [12] registered [ 0.301100] acpiphp: Slot [13] registered [ 0.303155] acpiphp: Slot [14] registered [ 0.304098] acpiphp: Slot [15] registered [ 0.306102] acpiphp: Slot [16] registered [ 0.308103] acpiphp: Slot [17] registered [ 0.309131] acpiphp: Slot [18] registered [ 0.311112] acpiphp: Slot [19] registered [ 0.313110] acpiphp: Slot [20] registered [ 0.314138] acpiphp: Slot [21] registered [ 0.316110] acpiphp: Slot [22] registered [ 0.318106] acpiphp: Slot [23] registered [ 0.319097] acpiphp: Slot [24] registered [ 0.321132] acpiphp: Slot [25] registered [ 0.323092] acpiphp: Slot [26] registered [ 0.324107] acpiphp: Slot [27] registered [ 0.326120] acpiphp: Slot [28] registered [ 0.327083] acpiphp: Slot [29] registered [ 0.329109] acpiphp: Slot [30] registered [ 0.330075] acpiphp: Slot [31] registered [ 0.331048] PCI host bridge to bus 0000:00 [ 0.333019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.336015] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.338026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.340024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.342022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.345041] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.347194] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.351043] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.354170] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.369892] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.375123] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.377020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.382022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.384017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.387441] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.389906] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.393097] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.396812] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.402017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.418025] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.423008] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.430556] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.436021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.444017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.467019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.480666] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.489018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.496014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.519020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.527796] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.542017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.550016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.566017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.576779] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.582027] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.591042] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.610019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.621619] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.635013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.646016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.678015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.687826] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.693011] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.700015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.721019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.736608] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.738638] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.741357] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.743367] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.745203] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.749033] iommu: Default domain type: Passthrough [ 0.750332] SCSI subsystem initialized [ 0.752170] ACPI: bus type USB registered [ 0.754097] usbcore: registered new interface driver usbfs [ 0.756079] usbcore: registered new interface driver hub [ 0.757105] usbcore: registered new device driver usb [ 0.759202] pps_core: LinuxPPS API ver. 1 registered [ 0.761010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.763045] PTP clock support registered [ 0.765129] EDAC MC: Ver: 3.0.0 [ 0.767516] PCI: Using ACPI for IRQ routing [ 0.769928] NetLabel: Initializing [ 0.771011] NetLabel: domain hash size = 128 [ 0.772007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.774078] NetLabel: unlabeled traffic allowed by default [ 0.776157] vgaarb: loaded [ 0.777289] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.779011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.784372] clocksource: Switched to clocksource kvm-clock [ 0.889285] VFS: Disk quotas dquot_6.6.0 [ 0.892753] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.895462] *** VALIDATE ramfs *** [ 0.896757] *** VALIDATE hugetlbfs *** [ 0.898539] pnp: PnP ACPI init [ 0.901256] pnp: PnP ACPI: found 6 devices [ 0.924800] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.928264] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.930632] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.933220] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.935948] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.938959] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.941938] NET: Registered protocol family 2 [ 0.944747] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.950441] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.954177] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.959627] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.963427] TCP: Hash tables configured (established 65536 bind 65536) [ 0.966128] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.968941] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.973084] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.976238] NET: Registered protocol family 1 [ 0.979406] RPC: Registered named UNIX socket transport module. [ 0.981656] RPC: Registered udp transport module. [ 0.983979] RPC: Registered tcp transport module. [ 0.985991] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.988471] NET: Registered protocol family 44 [ 0.990390] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.992703] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.995050] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.997790] PCI: CLS 0 bytes, default 64 [ 0.999572] Unpacking initramfs... [ 2.411693] debug: unmapping init [mem 0xffff9845bcc54000-0xffff9845bffbffff] [ 2.415789] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.417497] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.420513] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.915509] Initialise system trusted keyrings [ 2.917645] Key type blacklist registered [ 2.919835] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.930100] zbud: loaded [ 2.933154] *** VALIDATE nfs *** [ 2.934408] *** VALIDATE nfs4 *** [ 2.935882] pstore: using deflate compression [ 2.941600] Platform Keyring initialized [ 3.044409] NET: Registered protocol family 38 [ 3.046170] Key type asymmetric registered [ 3.047621] Asymmetric key parser 'x509' registered [ 3.049168] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.052112] io scheduler mq-deadline registered [ 3.054016] io scheduler kyber registered [ 3.055622] io scheduler bfq registered [ 3.058205] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.061353] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.063686] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.066669] ACPI: Power Button [PWRF] [ 3.072092] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.078493] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.091917] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.104109] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.124036] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.151619] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.181544] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.186722] Non-volatile memory driver v1.3 [ 3.188531] Linux agpgart interface v0.103 [ 3.218633] virtio_blk virtio1: [vda] 149768 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.221865] vda: detected capacity change from 0 to 76681216 [ 3.237986] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.241016] vdb: detected capacity change from 0 to 1073741824 [ 3.257062] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.260416] vdc: detected capacity change from 0 to 2621440000 [ 3.276155] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.279502] vdd: detected capacity change from 0 to 2621440000 [ 3.292983] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.295239] vde: detected capacity change from 0 to 4294967296 [ 3.311261] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.314373] vdf: detected capacity change from 0 to 4294967296 [ 3.324766] libphy: Fixed MDIO Bus: probed [ 3.330627] usbcore: registered new interface driver usbserial_generic [ 3.333022] usbserial: USB Serial support registered for generic [ 3.335707] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.339494] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.341458] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.344201] mousedev: PS/2 mouse device common for all mice [ 3.346845] rtc_cmos 00:05: RTC can wake from S4 [ 3.352189] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.352212] rtc_cmos 00:05: registered as rtc0 [ 3.356964] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.358856] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.359697] intel_pstate: CPU model not supported [ 3.365827] hid: raw HID events driver (C) Jiri Kosina [ 3.365886] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.367868] usbcore: registered new interface driver usbhid [ 3.367889] usbhid: USB HID core driver [ 3.368054] drop_monitor: Initializing network drop monitor service [ 3.368186] Initializing XFRM netlink socket [ 3.369447] NET: Registered protocol family 10 [ 3.384113] Segment Routing with IPv6 [ 3.385630] NET: Registered protocol family 17 [ 3.387512] mpls_gso: MPLS GSO support [ 3.393341] RAS: Correctable Errors collector initialized. [ 3.395113] AVX version of gcm_enc/dec engaged. [ 3.396599] AES CTR mode by8 optimization enabled [ 3.482211] sched_clock: Marking stable (3482185939, 0)->(4383860233, -901674294) [ 3.485137] registered taskstats version 1 [ 3.487223] Loading compiled-in X.509 certificates [ 3.490221] zswap: loaded using pool lzo/zbud [ 3.516259] Key type big_key registered [ 3.530814] Key type encrypted registered [ 3.533348] ima: No TPM chip found, activating TPM-bypass! [ 3.534977] ima: Allocated hash algorithm: sha1 [ 3.536187] ima: No architecture policies found [ 3.537401] evm: Initialising EVM extended attributes: [ 3.538988] evm: security.selinux [ 3.540231] evm: security.ima [ 3.541156] evm: security.capability [ 3.542575] evm: HMAC attrs: 0x1 [ 3.565360] rtc_cmos 00:05: setting system clock to 2026-09-05 20:02:53 UTC (1788638573) [ 3.570835] debug: unmapping init [mem 0xffffffff9ac03000-0xffffffff9adfffff] [ 3.574458] debug: unmapping init [mem 0xffffffff99982000-0xffffffff99c58fff] [ 3.583094] Write protecting the kernel read-only data: 28672k [ 3.586317] debug: unmapping init [mem 0xffffffff98003000-0xffffffff981fffff] [ 3.589287] debug: unmapping init [mem 0xffffffff98914000-0xffffffff989fffff] [ 3.624056] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.634279] systemd[1]: Detected virtualization kvm. [ 3.636588] systemd[1]: Detected architecture x86-64. [ 3.639060] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.666698] systemd[1]: No hostname configured. [ 3.668963] systemd[1]: Set hostname to . [ 3.671595] random: systemd: uninitialized urandom read (16 bytes read) [ 3.674679] systemd[1]: Initializing machine ID from random generator. [ 3.740866] random: ln: uninitialized urandom read (6 bytes read) [ 3.840706] random: systemd: uninitialized urandom read (16 bytes read) [ 3.844146] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.851277] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.856341] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.593595] device-mapper: uevent: version 1.0.3 [ 4.597111] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.468070] random: fast init done [ 5.483717] virtio_net virtio0 ens2: renamed from eth0 [ 5.513614] scsi host0: ata_piix [ 5.562829] scsi host1: ata_piix [ 5.564716] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.567310] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.559076] dracut-initqueue[583]: RTNETLINK answers: File exists [ 10.073139] random: crng init done [ 10.076379] random: 7 urandom warning(s) missed due to ratelimiting [ 11.475663] hrtimer: interrupt took 10652436 ns Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 12.066293] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.682939] printk: systemd: 24 output lines suppressed due to ratelimiting [ 16.650993] SELinux: Disabled at runtime. [ 16.822284] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 16.849933] systemd[1]: Detected virtualization kvm. [ 16.856097] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.806107] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.815784] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.827675] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.837135] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.843741] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.869720] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.882417] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. [ 19.094563] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Paths. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 20.546992] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.748343] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 21.806263] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 22.195358] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 22.342399] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit)[ 26.764404] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 27.536269] NFS: Registering the id_resolver key type [ 27.539693] Key type id_resolver registered [ 27.543995] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg442-server login: [ 84.807782] libcfs: loading out-of-tree module taints kernel. [ 84.886437] Key type ._llcrypt registered [ 84.890864] Key type .llcrypt registered [ 85.046225] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 104.912492] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 106.655667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 106.669215] alg: No test for adler32 (adler32-zlib) [ 108.097787] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 109.011787] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 110.769195] Key type lgssc registered [ 112.689661] Lustre: Echo OBD driver; http://www.lustre.org/ [ 130.413147] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 179.601507] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 193.184302] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 193.242699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 194.542090] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 194.583905] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 194.699170] Lustre: lustre-MDT0000: new disk, initializing [ 194.792289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 194.821324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 199.069860] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.711277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 212.783303] Lustre: 6530:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 212.845648] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 212.854156] Lustre: Skipped 1 previous similar message [ 212.963417] Lustre: lustre-MDT0001: new disk, initializing [ 213.046337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 213.081091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 213.095251] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 216.906531] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 221.710842] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 231.276608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 231.656398] Lustre: lustre-OST0000: new disk, initializing [ 231.660599] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 231.666819] Lustre: 8467:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 231.723206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 233.061120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 233.070759] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 233.170711] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 238.143906] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 253.636596] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 253.822761] Lustre: lustre-OST0001: new disk, initializing [ 253.827308] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 253.832794] Lustre: 9541:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 253.903431] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 259.087331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 259.097585] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 259.137179] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 260.295106] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 272.297271] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 279.430327] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 286.689698] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing check_logdir /tmp/testlogs/ [ 292.425814] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing yml_node [ 296.459572] Lustre: DEBUG MARKER: Client: 2.17.58.3 [ 299.397853] Lustre: DEBUG MARKER: MDS: 2.17.58.3 [ 301.875733] Lustre: DEBUG MARKER: OSS: 2.17.58.3 [ 303.676594] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 5 16:07:51 EDT 2026 [ 321.984510] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 323.423947] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 340.961182] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 340.963937] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 340.964441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 340.992179] Lustre: Skipped 1 previous similar message [ 345.771724] Lustre: server umount lustre-MDT0000 complete [ 350.672786] LustreError: 6522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788638920 with bad export cookie 4805056029905211348 [ 350.675632] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 350.690197] LustreError: 6522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 351.204622] LustreError: 7483:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 351.205367] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 351.211669] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 351.211689] Lustre: Skipped 3 previous similar messages [ 351.246848] LustreError: 7483:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 351.280357] Lustre: Skipped 3 previous similar messages [ 355.235975] LustreError: 10270:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 355.237646] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 355.253594] LustreError: 10270:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 355.282121] Lustre: Skipped 2 previous similar messages [ 357.143437] Lustre: server umount lustre-MDT0001 complete [ 361.035665] Lustre: server umount lustre-OST0000 complete [ 365.161121] Lustre: server umount lustre-OST0001 complete [ 371.490213] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 379.753212] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 415.995107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 416.185267] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 416.209596] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 416.284914] Lustre: lustre-MDT0000: new disk, initializing [ 416.363785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 416.385810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 421.002723] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 432.368186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 432.561722] Lustre: 15091:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 432.688443] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 432.701213] Lustre: Skipped 1 previous similar message [ 432.852436] Lustre: lustre-MDT0001: new disk, initializing [ 432.961882] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 433.029389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 433.039486] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 437.612466] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 446.719960] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 452.618860] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 459.438680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 459.768703] Lustre: lustre-OST0000: new disk, initializing [ 459.779157] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 459.789623] Lustre: 17059:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 459.905761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 461.361195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 461.376840] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 461.488398] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 465.896486] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 474.982785] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 479.396899] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 479.698152] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 486.373377] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 486.581147] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 492.520686] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 492.522114] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 492.524256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 492.554765] Lustre: Skipped 1 previous similar message [ 494.814823] Lustre: server umount lustre-OST0000 complete [ 499.681482] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 499.688696] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 499.703285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 499.712962] Lustre: Skipped 1 previous similar message [ 505.032579] Lustre: server umount lustre-MDT0000 complete [ 507.884268] LustreError: 15104:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 507.913346] LustreError: 15104:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 508.590037] LustreError: 15083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788639078 with bad export cookie 4805056029905212881 [ 508.610611] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 508.913224] Lustre: server umount lustre-MDT0001 complete [ 515.094457] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 16:11:22 (1788639082) [ 534.670570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 535.313919] LustreError: 19933:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 535.460554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 540.266707] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 540.647051] LustreError: 19934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 550.440044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 555.933957] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 565.069881] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 571.586905] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 573.683887] Lustre: Failing over lustre-MDT0000 [ 574.010505] Lustre: server umount lustre-MDT0000 complete [ 575.457779] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 575.470257] Lustre: Skipped 1 previous similar message [ 575.480113] LustreError: 20661:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 575.501056] LustreError: 20661:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 591.840473] Lustre: 3662:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788639145/real 1788639145] req@ffff984637b02a00 x1875523590576256/t0(0) o400->MGC192.168.204.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788639161 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 591.844542] LustreError: 19934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 591.880117] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 591.915116] LustreError: 19934:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 593.039321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 602.084321] LustreError: 3659:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff984503436d80 x1875523590579328/t0(0) o250->MGC192.168.204.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 602.454930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 602.460785] Lustre: Skipped 1 previous similar message [ 602.511283] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 607.105545] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 607.726588] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 607.730659] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 607.778564] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 617.106198] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 623.010392] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 623.033483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 623.040928] Lustre: Skipped 3 previous similar messages [ 635.360253] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 635.624904] Lustre: server umount lustre-MDT0000 complete [ 638.434332] LustreError: 19929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 638.450594] LustreError: 19929:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 639.606591] LustreError: 19914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788639209 with bad export cookie 4805056029905214295 [ 639.609710] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 639.623726] LustreError: 19914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 640.017201] Lustre: server umount lustre-MDT0001 complete [ 649.259996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 654.918392] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 663.518424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 668.198961] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 676.024484] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 681.675843] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 683.981543] Lustre: Failing over lustre-MDT0000 [ 684.016318] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 684.037146] Lustre: Skipped 1 previous similar message [ 684.050034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 684.057311] Lustre: Skipped 5 previous similar messages [ 684.348580] Lustre: server umount lustre-MDT0000 complete [ 703.460951] LustreError: 24002:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 703.480721] LustreError: 24002:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 704.480317] Lustre: 3661:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788639258/real 1788639258] req@ffff984502879500 x1875523590606080/t0(0) o400->MGC192.168.204.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788639274 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 704.519348] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 705.517340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 715.746921] LustreError: 3659:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff984637b01500 x1875523590609024/t0(0) o250->MGC192.168.204.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 716.118384] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 721.380059] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 721.387841] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 721.408671] Lustre: Skipped 1 previous similar message [ 721.423877] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 721.430871] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 731.321594] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 736.739575] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 736.750738] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 736.773085] Lustre: Skipped 1 previous similar message [ 736.804434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 741.490946] Lustre: server umount lustre-MDT0000 complete [ 744.983707] LustreError: 23254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788639314 with bad export cookie 4805056029905215233 [ 745.007103] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 745.365582] Lustre: server umount lustre-MDT0001 complete [ 754.995737] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 16:15:22 (1788639322) [ 772.309676] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 784.642648] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 827.964667] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 838.843530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 839.030287] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 839.069857] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 839.190851] Lustre: lustre-MDT0000: new disk, initializing [ 839.350838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 839.362302] Lustre: Skipped 3 previous similar messages [ 839.385922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 843.344854] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 853.932389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 854.037540] Lustre: 29653:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 854.419499] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 859.392462] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 869.505825] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 875.214890] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 882.406036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 882.717685] Lustre: 31621:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 884.829693] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 884.887415] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 884.961759] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 884.990125] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 889.554206] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 899.743077] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 911.399581] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 936.420551] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 936.437198] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 936.459754] Lustre: Skipped 2 previous similar messages [ 936.463390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 936.477969] Lustre: Skipped 3 previous similar messages [ 937.727227] Lustre: server umount lustre-OST0000 complete [ 946.658511] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 946.681818] LustreError: Skipped 1 previous similar message [ 956.384146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 956.540855] Lustre: server umount lustre-MDT0000 complete [ 956.899498] LustreError: 30966:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 956.916088] LustreError: 30966:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 960.593932] LustreError: 29644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788639530 with bad export cookie 4805056029905215639 [ 960.595881] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 960.613783] LustreError: 29644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 961.408043] Lustre: server umount lustre-MDT0001 complete [ 973.297739] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 976.727104] Key type lgssc unregistered [ 977.028522] LNet: 33600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 977.041423] LNetError: 33600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 977.065832] LNet: Removed LNI 192.168.204.142@tcp [ 978.127177] Key type .llcrypt unregistered [ 978.131207] Key type ._llcrypt unregistered [ 1003.678040] Key type ._llcrypt registered [ 1003.682268] Key type .llcrypt registered [ 1003.884482] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 1005.081421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1005.140413] alg: No test for adler32 (adler32-zlib) [ 1006.439273] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 1006.831274] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 1008.548782] Key type lgssc registered [ 1010.465073] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1020.392727] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1020.413590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1021.930169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1027.187410] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1031.162721] LustreError: 35007:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1031.173799] LustreError: 35007:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1035.766635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1036.058720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1040.288796] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1047.581996] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1053.127871] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1060.738037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1061.241029] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1068.898213] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1077.662990] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1080.767478] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 1094.500417] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 16:21:02 (1788639662) [ 1096.728716] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1098.845760] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1106.292111] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 16:21:13 (1788639673) [ 1114.940111] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 16:21:22 (1788639682) [ 1115.849707] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1125.046319] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1125.053703] Lustre: Skipped 2 previous similar messages [ 1126.703258] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 16:21:34 (1788639694) [ 1130.180433] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1130.185264] Lustre: Skipped 3 previous similar messages [ 1151.383326] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1151.391164] Lustre: Skipped 6 previous similar messages [ 1155.141901] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 16:22:01 (1788639721) [ 1159.140226] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1163.443539] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1173.295486] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 16:22:20 (1788639740) [ 1175.446358] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1175.453666] Lustre: Skipped 3 previous similar messages [ 1188.931351] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 16:22:36 (1788639756) [ 1204.717649] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1204.731506] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1204.748896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1208.801622] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1208.810888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1208.812034] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1208.828067] Lustre: Skipped 1 previous similar message [ 1210.442880] Lustre: server umount lustre-OST0000 complete [ 1215.458300] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1215.473148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1215.503097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1219.048852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1219.050710] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1224.168370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1224.183399] Lustre: Skipped 2 previous similar messages [ 1228.257651] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1228.440801] Lustre: server umount lustre-MDT0000 complete [ 1231.898526] LustreError: 34989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788639801 with bad export cookie 13083698860721661015 [ 1231.905338] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1231.911307] LustreError: 34989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1232.470799] Lustre: server umount lustre-MDT0001 complete [ 1243.323921] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 1245.865887] Key type lgssc unregistered [ 1246.160712] LNet: 42871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1246.173674] LNetError: 42871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1246.192990] LNet: Removed LNI 192.168.204.142@tcp [ 1246.909143] Key type .llcrypt unregistered [ 1246.912288] Key type ._llcrypt unregistered [ 1263.796183] Key type ._llcrypt registered [ 1263.800946] Key type .llcrypt registered [ 1263.903741] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 1277.297301] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 1278.444796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1278.488988] alg: No test for adler32 (adler32-zlib) [ 1279.618467] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 1279.809669] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 1281.488213] Key type lgssc registered [ 1282.740518] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1324.320447] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1324.377105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1325.791387] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1325.847257] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1325.984885] Lustre: lustre-MDT0000: new disk, initializing [ 1326.103633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1326.142549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1331.451688] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1342.724554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1342.892562] Lustre: 46011:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1342.948794] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1342.952765] Lustre: Skipped 1 previous similar message [ 1343.075808] Lustre: lustre-MDT0001: new disk, initializing [ 1343.185496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1343.225858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1343.230967] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1348.436617] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1359.315419] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1366.766775] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1375.966983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1376.295915] Lustre: lustre-OST0000: new disk, initializing [ 1376.301395] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1376.316657] Lustre: 47981:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1376.439581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1384.124649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1384.135340] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1384.346513] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1386.136733] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1397.440439] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1403.347598] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1403.600822] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1408.301711] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1408.475564] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1415.141201] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1415.143602] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1415.159987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1415.193749] Lustre: Skipped 1 previous similar message [ 1416.888022] Lustre: server umount lustre-OST0000 complete [ 1425.377524] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1425.383221] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1425.429833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1425.449945] LustreError: Skipped 1 previous similar message [ 1425.475071] Lustre: Skipped 2 previous similar messages [ 1428.265438] Lustre: server umount lustre-MDT0000 complete [ 1433.269810] LustreError: 46004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788640003 with bad export cookie 8563349844430913442 [ 1433.277654] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1433.285416] LustreError: 46004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1433.689480] Lustre: server umount lustre-MDT0001 complete [ 1448.684662] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 16:26:54 (1788640014) [ 1459.007946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1459.909566] LustreError: 50443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1459.946029] LustreError: 50443:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1460.075511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1465.320877] LustreError: 50444:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1465.350824] LustreError: 50444:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1465.603644] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1469.411304] LustreError: 50443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1474.547823] LustreError: 50444:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1477.029339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1477.544475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1482.854956] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1492.847910] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1499.850534] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1507.429079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1507.766515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1513.643687] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1520.895482] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1535.704433] LustreError: 52994:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1536.505778] Lustre: Modifying parameter general.timeout=1 in log params [ 1538.004787] Lustre: Modifying parameter general.timeout=3 in log params [ 1538.016401] Lustre: Skipped 1 previous similar message [ 1540.214612] Lustre: Modifying parameter general.timeout=6 in log params [ 1540.219245] Lustre: Skipped 2 previous similar messages [ 1544.404261] Lustre: Modifying parameter general.timeout=11 in log params [ 1544.417517] Lustre: Skipped 4 previous similar messages [ 1553.281461] Lustre: Modifying parameter general.timeout=20 in log params [ 1553.286346] Lustre: Skipped 8 previous similar messages [ 1570.477079] Lustre: Modifying parameter general.timeout=34 in log params [ 1570.485391] Lustre: Skipped 13 previous similar messages [ 1602.703562] Lustre: Modifying parameter general.timeout=67 in log params [ 1602.712436] Lustre: Skipped 32 previous similar messages [ 1667.407979] Lustre: Modifying parameter general.timeout=128 in log params [ 1667.414884] Lustre: Skipped 60 previous similar messages [ 1680.418936] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 16:30:47 (1788640247) [ 1684.451960] Lustre: 59948:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1686.562589] Lustre: 60060:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1691.209694] Lustre: 60257:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1691.218364] Lustre: 60257:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1695.857365] Lustre: 60459:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1695.869459] Lustre: 60459:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1728.606304] Lustre: 61828:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1728.640292] LustreError: 61828:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1734.300885] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 16:31:41 (1788640301) [ 1754.083308] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1754.092810] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1754.104139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1754.110156] Lustre: Skipped 3 previous similar messages [ 1754.118734] Lustre: Skipped 2 previous similar messages [ 1756.455165] Lustre: server umount lustre-MDT0000 complete [ 1759.203511] LustreError: 51877:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1759.221683] LustreError: 51877:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1760.458318] LustreError: 53044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788640330 with bad export cookie 8563349844430914324 [ 1760.470562] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1760.473592] LustreError: 53044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1760.827274] Lustre: server umount lustre-MDT0001 complete [ 1775.118203] Lustre: server umount lustre-OST0000 complete [ 1806.492771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1806.878447] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1806.917379] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1806.930899] Lustre: Skipped 24 previous similar messages [ 1807.184630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1812.916792] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1824.408849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1824.576206] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1824.629975] Lustre: 64364:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1830.251498] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1838.910947] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1845.756353] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1853.103098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1853.295536] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1853.554353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1853.564266] Lustre: Skipped 1 previous similar message [ 1861.029088] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1869.509419] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1871.490784] Lustre: 66933:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1883.645589] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1883.655280] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1883.658423] LustreError: Skipped 1 previous similar message [ 1883.659476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1883.672159] Lustre: Skipped 1 previous similar message [ 1883.704395] Lustre: Skipped 1 previous similar message [ 1884.835744] Lustre: server umount lustre-OST0000 complete [ 1891.810134] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1891.818905] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1891.864745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1896.120383] Lustre: server umount lustre-MDT0000 complete [ 1898.979246] LustreError: 64371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1899.013960] LustreError: 64371:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1901.725937] LustreError: 64356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788640471 with bad export cookie 8563349844430917558 [ 1901.736252] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1901.753540] LustreError: 64356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1902.074793] Lustre: server umount lustre-MDT0001 complete [ 1917.283605] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 1920.386940] Key type lgssc unregistered [ 1920.751910] LNet: 68143:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1920.769512] LNetError: 68143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1920.830085] LNet: Removed LNI 192.168.204.142@tcp [ 1921.891432] Key type .llcrypt unregistered [ 1921.893558] Key type ._llcrypt unregistered [ 1953.080380] Key type ._llcrypt registered [ 1953.089888] Key type .llcrypt registered [ 1953.302135] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 1955.098764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1955.166347] alg: No test for adler32 (adler32-zlib) [ 1956.220562] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 1956.454180] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 1958.144300] Key type lgssc registered [ 1959.692788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1970.721135] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1970.765968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1972.886916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1980.236513] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1982.515039] LustreError: 69554:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1982.563632] LustreError: 69554:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1987.566595] LustreError: 69555:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1992.685474] LustreError: 69554:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1994.189923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1995.823226] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2002.622137] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2011.285889] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2017.437415] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2026.922474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2027.945229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2037.769401] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2046.860712] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2055.875560] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2066.913440] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2066.939354] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2066.968130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2067.946048] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2067.948456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2067.993238] Lustre: Skipped 2 previous similar messages [ 2072.955367] Lustre: server umount lustre-MDT0000 complete [ 2073.058341] LustreError: 69554:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2073.093057] LustreError: 69554:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2077.807628] LustreError: 69535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788640647 with bad export cookie 1815857889142662592 [ 2077.833813] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2078.179301] LustreError: 71404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2078.189959] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2078.203632] Lustre: Skipped 1 previous similar message [ 2078.578866] Lustre: server umount lustre-MDT0001 complete [ 2085.984097] Lustre: server umount lustre-OST0000 complete [ 2117.060060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2117.497457] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2117.546240] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2117.940902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2123.048695] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2134.781957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2134.905697] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2134.964254] Lustre: 74428:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2135.632175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2140.896172] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2149.511638] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2157.530339] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2165.231100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2165.468753] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2165.977164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2172.632651] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2181.262385] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2183.678251] Lustre: 77004:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2186.077777] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2196.964541] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2196.972682] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2196.995971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2199.586603] Lustre: server umount lustre-OST0000 complete [ 2207.204241] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2207.214607] LustreError: Skipped 1 previous similar message [ 2207.227531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2207.260727] Lustre: Skipped 1 previous similar message [ 2207.278305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2207.288123] Lustre: Skipped 2 previous similar messages [ 2210.891225] Lustre: server umount lustre-MDT0000 complete [ 2212.839325] LustreError: 74439:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2212.867878] LustreError: 74439:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2215.741275] LustreError: 75185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788640785 with bad export cookie 1815857889142663439 [ 2215.755574] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2215.757066] LustreError: 75185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2216.266477] Lustre: server umount lustre-MDT0001 complete [ 2230.329387] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 2233.529487] Key type lgssc unregistered [ 2233.823536] LNet: 78218:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2233.832639] LNetError: 78218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2233.845280] LNet: Removed LNI 192.168.204.142@tcp [ 2234.949635] Key type .llcrypt unregistered [ 2234.956119] Key type ._llcrypt unregistered [ 2265.618695] Key type ._llcrypt registered [ 2265.630445] Key type .llcrypt registered [ 2265.768241] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 2267.634153] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2267.674364] alg: No test for adler32 (adler32-zlib) [ 2269.020512] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 2269.420936] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 2271.168672] Key type lgssc registered [ 2273.044761] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2284.486134] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2284.531344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2286.273601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2293.114568] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2296.802447] LustreError: 79634:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2296.836708] LustreError: 79634:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2301.936485] LustreError: 79635:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2305.199360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2306.303922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2312.538693] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2321.461859] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2329.257488] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2337.072434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2337.890830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2345.563450] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2354.809896] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2364.874706] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2365.886641] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2367.257357] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2369.850559] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2369.858190] Lustre: Skipped 2 previous similar messages [ 2374.670119] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2374.680678] Lustre: Skipped 4 previous similar messages [ 2383.285596] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2383.292536] Lustre: Skipped 7 previous similar messages [ 2399.485289] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2399.488887] Lustre: Skipped 13 previous similar messages [ 2432.183970] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2432.196141] Lustre: Skipped 31 previous similar messages [ 2478.773457] Lustre: 87738:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2481.956984] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 16:44:09 (1788641049) [ 2483.668250] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2485.723080] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 16:44:13 (1788641053) [ 2493.409422] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2493.422160] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2493.448211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2494.435629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2494.438609] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2494.446167] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2496.232350] Lustre: server umount lustre-OST0000 complete [ 2504.677700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2504.682337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2504.693152] Lustre: Skipped 1 previous similar message [ 2504.706614] Lustre: Skipped 2 previous similar messages [ 2507.110958] Lustre: server umount lustre-MDT0000 complete [ 2509.799819] LustreError: 79635:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2509.824214] LustreError: 79635:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2511.903132] LustreError: 79615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788641081 with bad export cookie 13860468762885227185 [ 2511.908956] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2511.928416] LustreError: 79615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2512.570807] Lustre: server umount lustre-MDT0001 complete [ 2526.532638] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 2530.877423] Key type lgssc unregistered [ 2531.187902] LNet: 88988:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2531.201785] LNetError: 88988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2531.241428] LNet: Removed LNI 192.168.204.142@tcp [ 2532.512073] Key type .llcrypt unregistered [ 2532.513920] Key type ._llcrypt unregistered [ 2537.984495] Key type ._llcrypt registered [ 2537.989742] Key type .llcrypt registered [ 2538.202838] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2545.120248] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules [ 2546.580244] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2546.627105] alg: No test for adler32 (adler32-zlib) [ 2548.146054] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 2548.696305] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 2550.528783] Key type lgssc registered [ 2551.731754] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2553.074379] LustreError: 90226:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2573.551945] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 2575.786408] LustreError: 90226:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2583.721883] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2583.772780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2585.313041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2590.337807] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2595.811513] LustreError: 91258:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2595.828048] LustreError: 91258:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2599.132701] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 16:46:06 (1788641166) [ 2600.935512] LustreError: 91257:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2604.664838] LustreError: 91289:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2604.690695] LustreError: 91289:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2604.702370] LustreError: 91289:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 2605.221298] Lustre: server umount lustre-MDT0000 complete [ 2621.253493] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 2624.113176] Key type lgssc unregistered [ 2624.499137] LNet: 92780:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2624.514627] LNetError: 92780:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2625.577985] LNet: Removed LNI 192.168.204.142@tcp [ 2626.469419] Key type .llcrypt unregistered [ 2626.473761] Key type ._llcrypt unregistered [ 2645.658723] Key type ._llcrypt registered [ 2645.663158] Key type .llcrypt registered [ 2645.840299] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 2646.679278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2646.723699] alg: No test for adler32 (adler32-zlib) [ 2647.856670] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 2648.128353] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 2649.792186] Key type lgssc registered [ 2651.067496] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2660.081364] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2660.127406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2661.612124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2667.644402] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2671.077963] LustreError: 94003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2671.104918] LustreError: 94003:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2676.195943] LustreError: 94004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2677.953133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2678.759089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2683.660418] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2692.527511] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2698.988107] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2706.437565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2707.116975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2713.233379] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2721.334660] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2780.231515] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 16:49:07 (1788641347) [ 2782.304661] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2784.369510] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 16:49:11 (1788641351) [ 2790.885441] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2790.894611] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2790.931755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2793.454522] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2793.458700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2793.482387] Lustre: Skipped 1 previous similar message [ 2793.511269] Lustre: Skipped 2 previous similar messages [ 2794.932543] Lustre: server umount lustre-MDT0000 complete [ 2798.568205] LustreError: 94410:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2798.594547] LustreError: 94410:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2799.411044] LustreError: 94746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788641369 with bad export cookie 10729035321382131280 [ 2799.414073] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2799.434370] LustreError: 94746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2799.855784] Lustre: server umount lustre-MDT0001 complete [ 2816.020235] Lustre: server umount lustre-OST0000 complete [ 2831.575897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2832.259986] LustreError: 98323:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2832.454854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2837.477845] LustreError: 98324:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2837.989464] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2847.447683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2847.713858] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2848.340720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2853.189293] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2862.356508] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2868.133890] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2885.536707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2885.891627] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2885.899064] LustreError: 98311:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2885.951455] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2885.967232] LustreError: 100408:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2885.981260] LustreError: 100408:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2885.986723] LustreError: 100408:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2886.107396] Lustre: server umount lustre-OST0000 complete [ 2886.122843] LustreError: 100408:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2897.741200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2898.069130] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2898.080295] LustreError: 98313:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2898.109451] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2898.117277] LustreError: 100874:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2898.126582] LustreError: 100874:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2898.137082] LustreError: 100874:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2898.215491] Lustre: server umount lustre-OST0000 complete [ 2898.224491] LustreError: 100874:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2909.707393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2910.006405] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2910.011736] Lustre: Found index 0 for lustre-OST0000, updating log [ 2910.019631] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2910.183605] Lustre: lustre-OST0000: new disk, initializing [ 2910.202962] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2910.224847] Lustre: 101409:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2910.717325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2917.766696] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2922.398086] Lustre: Failing over lustre-OST0000 [ 2922.584648] Lustre: server umount lustre-OST0000 complete [ 2926.502981] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2926.517543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2926.527236] Lustre: Skipped 1 previous similar message [ 2926.547964] Lustre: Skipped 1 previous similar message [ 2931.685968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2931.697754] Lustre: Skipped 1 previous similar message [ 2932.933283] Lustre: server umount lustre-MDT0000 complete [ 2936.803195] LustreError: 98323:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2936.825459] LustreError: 98323:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2936.869825] LustreError: 100891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788641506 with bad export cookie 10729035321382134024 [ 2936.873041] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2936.879162] LustreError: 100891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2947.236386] Lustre: server umount lustre-MDT0001 complete [ 2957.336514] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 16:52:04 (1788641524) [ 2974.274338] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 2985.324882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2985.727985] LustreError: 103757:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2985.828990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2990.841942] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3000.168061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3006.162721] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3009.252671] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3016.324069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3017.035080] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3023.612206] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3025.203366] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3025.237854] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3025.239491] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3032.855243] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3032.989277] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3033.088611] Lustre: lustre-OST0001: new disk, initializing [ 3033.098611] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3033.112719] Lustre: 106076:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3033.404821] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3033.410127] Lustre: Skipped 2 previous similar messages [ 3040.198836] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3042.957188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3042.977435] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3043.095021] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3048.317682] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3056.702158] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3062.759237] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3062.772505] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3062.794325] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3067.096391] Lustre: server umount lustre-MDT0001 complete [ 3068.681852] LustreError: 105272:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3068.720296] LustreError: 105272:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3077.409710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3077.493539] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3077.500463] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3077.532723] Lustre: 103745:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3078.639449] LustreError: 107604:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3078.646966] LustreError: 107604:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3078.653119] LustreError: 103749:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3078.764770] LustreError: 107619:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.142@tcp: cfg command failed: rc = -17 [ 3078.774962] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3078.774962] [ 3083.000972] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3083.763666] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3092.145350] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 16:54:19 (1788641659) [ 3110.467309] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 3128.546379] Lustre: 109691:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3150.999445] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3155.684478] Lustre: 110826:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3252.200932] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3252.209164] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3252.224191] Lustre: Skipped 2 previous similar messages [ 3252.228834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3252.236689] Lustre: Skipped 4 previous similar messages [ 3255.121913] Lustre: server umount lustre-MDT0000 complete [ 3257.826183] LustreError: 103754:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3257.863863] LustreError: 103754:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 3259.473515] LustreError: 105275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788641829 with bad export cookie 10729035321382135025 [ 3259.483407] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3259.495745] LustreError: 105275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3260.282146] Lustre: server umount lustre-MDT0001 complete [ 3275.748873] Lustre: server umount lustre-OST0000 complete [ 3278.692815] Lustre: 93315:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788641832/real 1788641832] req@ffff984636c6b480 x1875526254176384/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788641848 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3278.717479] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3278.728292] Lustre: Skipped 3 previous similar messages [ 3280.072517] Lustre: server umount lustre-OST0001 complete [ 3287.116527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3298.430894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3309.302950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3322.006257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3344.302657] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 3355.493603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3355.511450] Lustre: lustre-MDT0000: reset Object Index mappings [ 3355.899741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3355.910691] Lustre: Skipped 1 previous similar message [ 3359.816537] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3368.679620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3368.710703] Lustre: lustre-MDT0001: reset Object Index mappings [ 3374.190455] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3377.419415] Lustre: 115928:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3384.292181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3391.155989] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3393.272922] LustreError: 116280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3393.308199] LustreError: 116280:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3393.311830] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:231 to 0x280000400:257) [ 3400.708046] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 3401.351602] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3407.383672] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:157 to 0x2c0000401:193) [ 3407.385698] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3410.162609] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3420.169424] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3424.234847] Lustre: 117804:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3482.119278] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 17:00:49 (1788642049) [ 3507.169748] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3507.176548] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3507.193472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3507.199926] Lustre: Skipped 3 previous similar messages [ 3512.516600] Lustre: server umount lustre-MDT0000 complete [ 3516.441364] LustreError: 114762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788642086 with bad export cookie 10729035321382181911 [ 3516.446787] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3516.457141] LustreError: 114762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3523.038496] Lustre: server umount lustre-MDT0001 complete [ 3527.100672] LustreError: 118579:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x280000401:0x133:0x0].0x0 (ffff98450b038c00) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 3533.349879] Lustre: server umount lustre-OST0000 complete [ 3542.918326] Lustre: server umount lustre-OST0001 complete [ 3550.187487] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 3561.462301] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 3595.945042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3596.187432] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3596.205439] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3596.270934] Lustre: lustre-MDT0000: new disk, initializing [ 3596.345878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3596.353438] Lustre: Skipped 3 previous similar messages [ 3596.376550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3601.058288] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3608.512220] Lustre: server umount lustre-MDT0000 complete [ 3618.235306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3618.493609] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3618.510117] Lustre: 122089:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3618.522961] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3623.910504] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3629.981342] Lustre: server umount lustre-MDT0000 complete [ 3638.047451] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 17:03:25 (1788642205) [ 3639.636353] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3641.335537] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 17:03:29 (1788642209) [ 3660.016983] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 3671.558947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3676.259243] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3686.277778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3686.441726] Lustre: 124140:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3686.516108] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3686.522103] Lustre: Skipped 1 previous similar message [ 3686.586284] Lustre: lustre-MDT0001: new disk, initializing [ 3686.717388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3686.722363] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3692.283515] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3698.832275] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3706.277087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3706.521742] Lustre: lustre-OST0000: new disk, initializing [ 3706.524832] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3706.530708] Lustre: 125714:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3708.472843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3708.505486] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3708.620467] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3713.788837] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3725.745365] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3725.938558] Lustre: lustre-OST0001: new disk, initializing [ 3725.942463] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3725.950558] Lustre: 126587:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3727.974526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3727.986664] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3728.109624] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3733.564954] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3744.388560] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3748.690269] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4159.714993] Lustre: 133006:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4159.724616] Lustre: 133006:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4159.732780] Lustre: 133006:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4159.739748] Lustre: 133006:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4159.748233] Lustre: 133006:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4159.753985] Lustre: 133006:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4484.537694] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 17:17:31 (1788643051) [ 4500.969381] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4500.978909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4501.005343] Lustre: Skipped 5 previous similar messages [ 4501.010181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4501.016672] Lustre: Skipped 8 previous similar messages [ 4511.724886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4511.733696] Lustre: Skipped 11 previous similar messages [ 4515.297554] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4515.707599] Lustre: server umount lustre-MDT0000 complete [ 4516.837263] LustreError: 126499:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4516.860359] LustreError: 126499:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 4519.705063] LustreError: 133029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788643089 with bad export cookie 10729035321382251225 [ 4519.708849] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4519.719736] LustreError: 133029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4521.955361] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4521.966521] Lustre: Skipped 3 previous similar messages [ 4530.096162] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4530.105037] Lustre: Skipped 4 previous similar messages [ 4534.240241] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4534.843434] Lustre: server umount lustre-MDT0001 complete [ 4554.208969] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4554.492294] Lustre: server umount lustre-OST0000 complete [ 4565.874849] Lustre: server umount lustre-OST0001 complete [ 4574.209960] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 4585.259376] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 4624.448714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4624.747639] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4624.783107] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4624.905290] Lustre: lustre-MDT0000: new disk, initializing [ 4624.972332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4624.978842] Lustre: Skipped 5 previous similar messages [ 4625.006120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4629.531235] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4640.411939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4640.599835] Lustre: 136074:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4640.659172] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4640.663319] Lustre: Skipped 1 previous similar message [ 4640.774661] Lustre: lustre-MDT0001: new disk, initializing [ 4640.989725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4641.021494] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4646.127350] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4656.826472] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4662.349031] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4669.675574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4670.128510] Lustre: lustre-OST0000: new disk, initializing [ 4670.135639] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4670.147893] Lustre: 138043:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4670.277591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4670.284679] Lustre: Skipped 1 previous similar message [ 4671.817588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4671.832203] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4671.975259] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4676.980400] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4686.624151] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4697.126525] Lustre: lustre-MDD0000: changelog on [ 4699.842976] Lustre: lustre-MDD0001: changelog on [ 6884.153261] Lustre: lustre-MDD0001: changelog off [ 6886.320875] Lustre: 139496:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6887.426409] Lustre: lustre-MDD0000: changelog off [ 6898.661264] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6898.668884] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6898.693345] Lustre: Skipped 1 previous similar message [ 6898.704635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6899.172759] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6900.748841] Lustre: server umount lustre-OST0000 complete [ 6908.898448] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6908.920101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6908.952234] Lustre: Skipped 1 previous similar message [ 6908.974202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6908.984387] Lustre: Skipped 2 previous similar messages [ 6919.137532] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6919.544812] Lustre: server umount lustre-MDT0000 complete [ 6919.655749] LustreError: 136953:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6919.664287] LustreError: 136953:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 6924.781090] LustreError: 136083:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6924.823967] LustreError: 136083:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6924.914157] LustreError: 136068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788645494 with bad export cookie 10729035321383069875 [ 6924.926130] LustreError: 136068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6924.941150] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6925.572971] Lustre: server umount lustre-MDT0001 complete [ 6937.276780] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 6940.977932] Key type lgssc unregistered [ 6941.490085] LNet: 147233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6941.517444] LNetError: 147233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6941.542492] LNet: Removed LNI 192.168.204.142@tcp [ 6942.740179] Key type .llcrypt unregistered [ 6942.742089] Key type ._llcrypt unregistered [ 6969.849756] Key type ._llcrypt registered [ 6969.854838] Key type .llcrypt registered [ 6970.000545] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 6990.189692] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 6992.061138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6992.193137] alg: No test for adler32 (adler32-zlib) [ 6993.374865] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 6993.898191] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 6995.664154] Key type lgssc registered [ 6997.032405] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7038.533433] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7038.562591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7039.863444] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7039.898313] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7039.991843] Lustre: lustre-MDT0000: new disk, initializing [ 7040.120584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7040.154532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7045.041847] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7056.316780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7056.524411] Lustre: 150432:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7056.577392] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7056.581499] Lustre: Skipped 1 previous similar message [ 7056.648719] Lustre: lustre-MDT0001: new disk, initializing [ 7056.797889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7056.853794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7056.868113] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7061.745881] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7073.627368] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7082.063457] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7089.977375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7090.328228] Lustre: lustre-OST0000: new disk, initializing [ 7090.332502] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7090.344695] Lustre: 152403:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7090.460497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7097.974899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7097.991800] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7098.117178] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7098.343369] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7108.372960] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7113.550183] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_module obdecho/obdecho [ 7115.442545] LustreError: 153418:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7115.450681] LustreError: 153418:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7117.834543] LustreError: 153514:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 7133.665688] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7133.683937] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7133.715754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7134.184892] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7134.196567] Lustre: Skipped 1 previous similar message [ 7139.299627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7139.324925] Lustre: Skipped 5 previous similar messages [ 7140.164729] Lustre: server umount lustre-MDT0000 complete [ 7144.541849] LustreError: 152402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788645714 with bad export cookie 15684609091733273361 [ 7144.556704] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7144.566957] LustreError: 152402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7145.127824] Lustre: server umount lustre-MDT0001 complete [ 7160.140979] Lustre: server umount lustre-OST0000 complete [ 7185.023469] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 7196.212219] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 7235.338824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7235.579971] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7235.601907] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7235.657528] Lustre: lustre-MDT0000: new disk, initializing [ 7235.725891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7235.741755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7239.866938] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7253.940860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7254.045775] Lustre: 157232:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7254.128362] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7254.136422] Lustre: Skipped 1 previous similar message [ 7254.266197] Lustre: lustre-MDT0001: new disk, initializing [ 7254.369147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7254.425477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7254.441133] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7260.876788] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7271.706029] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7277.253373] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7284.578139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7285.161163] Lustre: lustre-OST0000: new disk, initializing [ 7285.171258] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7285.190874] Lustre: 159201:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7285.345222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7290.497994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7290.529223] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7290.733570] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7293.469295] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7308.001727] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7312.096498] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7312.314587] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7317.957214] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7318.233654] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7321.579540] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7321.581982] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7321.608156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7321.628950] Lustre: Skipped 1 previous similar message [ 7321.640458] Lustre: Skipped 1 previous similar message [ 7326.706901] Lustre: server umount lustre-OST0000 complete [ 7331.297220] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7331.305167] LustreError: Skipped 1 previous similar message [ 7331.309630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7331.327782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7336.931061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7336.944661] Lustre: Skipped 2 previous similar messages [ 7337.630462] Lustre: server umount lustre-MDT0000 complete [ 7341.598760] LustreError: 157224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788645911 with bad export cookie 15684609091733274313 [ 7341.604292] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7341.620834] LustreError: 157224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7341.978565] Lustre: server umount lustre-MDT0001 complete [ 7346.397122] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 18:05:13 (1788645913) [ 7348.256104] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7350.282719] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 18:05:17 (1788645917) [ 7358.315497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7358.681802] LustreError: 161616:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7358.707909] LustreError: 161616:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7358.782553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7363.739629] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7364.081158] LustreError: 161617:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7368.227971] LustreError: 161616:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7373.289459] LustreError: 161617:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7374.245220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7380.848599] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7390.103021] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7396.218470] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7403.990538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7404.367171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7404.386668] Lustre: Skipped 1 previous similar message [ 7412.868238] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7422.016979] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7441.382604] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7441.395594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7441.413478] Lustre: Skipped 1 previous similar message [ 7441.417484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7445.294319] Lustre: server umount lustre-MDT0000 complete [ 7448.809728] LustreError: 161597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788646018 with bad export cookie 15684609091733275202 [ 7448.815511] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7448.822364] LustreError: 161597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7448.824126] LustreError: 163459:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7449.570993] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7449.586624] Lustre: Skipped 2 previous similar messages [ 7458.730293] LustreError: 163459:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7458.733345] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7458.764520] LustreError: 163459:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7458.783497] Lustre: Skipped 8 previous similar messages [ 7463.392192] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7463.745258] Lustre: server umount lustre-MDT0001 complete [ 7470.403049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7485.045472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7485.408716] LustreError: 165481:0:(llog.c:1655:llog_backup()) MGC192.168.204.142@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7485.420286] Lustre: 165481:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.142@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7488.484173] LustreError: 165486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7488.515732] LustreError: 165486:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7523.300343] LustreError: 165488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7523.334342] LustreError: 165488:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 27 previous similar messages [ 7546.656864] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7546.669245] Lustre: 165481:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.142@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7587.323882] LustreError: 165487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7587.330739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7587.354709] LustreError: 165487:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 42 previous similar messages [ 7592.141327] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7592.431621] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7601.963707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7613.930063] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7619.057566] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xd9aaecde499fdb9b [ 7619.075137] Lustre: MGC192.168.204.142@tcp: Connection restored to 0@lo (at 0@lo) [ 7623.791080] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7624.691473] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7632.052029] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7634.043358] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7639.858603] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7641.652792] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7665.646170] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7665.648264] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7665.682067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7665.682081] Lustre: Skipped 1 previous similar message [ 7669.432555] Lustre: server umount lustre-MDT0000 complete [ 7673.014371] LustreError: 163462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788646242 with bad export cookie 15684609091733281691 [ 7673.017296] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7673.023539] LustreError: 163462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7673.276893] Lustre: server umount lustre-MDT0001 complete [ 7686.293241] Lustre: server umount lustre-OST0000 complete [ 7711.155758] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 7721.659595] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 7765.312993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7765.633118] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7765.724410] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7765.881676] Lustre: lustre-MDT0000: new disk, initializing [ 7765.947282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7765.953136] Lustre: Skipped 1 previous similar message [ 7765.963613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7771.540378] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7786.121532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7786.308233] Lustre: 170596:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7786.529521] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7792.956689] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7803.312969] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7809.647936] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7817.283100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7817.488610] Lustre: lustre-OST0000: new disk, initializing [ 7817.491793] Lustre: Skipped 1 previous similar message [ 7817.495721] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7817.499650] Lustre: Skipped 2 previous similar messages [ 7817.503979] Lustre: 172569:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7824.327651] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7825.993892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7826.005029] Lustre: Skipped 1 previous similar message [ 7826.012038] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7826.170487] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7835.588229] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7841.042314] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7841.215723] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7845.764550] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7846.055939] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7852.002928] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7852.021194] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7852.048128] Lustre: Skipped 2 previous similar messages [ 7852.061560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7852.070015] Lustre: Skipped 3 previous similar messages [ 7862.240479] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7862.419606] Lustre: server umount lustre-OST0000 complete [ 7873.009921] Lustre: server umount lustre-MDT0000 complete [ 7876.578332] LustreError: 170609:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7876.592460] LustreError: 170609:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 31 previous similar messages [ 7877.285965] LustreError: 170589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788646447 with bad export cookie 15684609091733298491 [ 7877.287774] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7877.297972] LustreError: 170589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7877.663184] Lustre: server umount lustre-MDT0001 complete [ 7887.096757] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 18:14:14 (1788646454) [ 7894.288101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7900.032623] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7911.432296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7917.928343] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7926.417272] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7933.284536] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7940.251199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7947.547206] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7955.671892] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7977.709723] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 18:15:45 (1788646545) [ 7982.049885] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7982.058495] LustreError: Skipped 1 previous similar message [ 7982.062706] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7982.079467] Lustre: Skipped 3 previous similar messages [ 7982.090138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7982.098614] Lustre: Skipped 7 previous similar messages [ 7987.965793] Lustre: server umount lustre-OST0000 complete [ 7998.595243] Lustre: server umount lustre-MDT0000 complete [ 8002.559225] LustreError: 175721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788646572 with bad export cookie 15684609091733299373 [ 8002.566249] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8002.576576] LustreError: 175721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8002.924221] Lustre: server umount lustre-MDT0001 complete [ 8014.956512] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 8018.297202] Key type lgssc unregistered [ 8018.641291] LNet: 178737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8018.647197] LNetError: 178737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8018.665337] LNet: Removed LNI 192.168.204.142@tcp [ 8019.558556] Key type .llcrypt unregistered [ 8019.563974] Key type ._llcrypt unregistered [ 8036.716736] Key type ._llcrypt registered [ 8036.719755] Key type .llcrypt registered [ 8036.824586] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 8037.831142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8037.863238] alg: No test for adler32 (adler32-zlib) [ 8038.833871] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 8039.150893] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 8040.824729] Key type lgssc registered [ 8041.910534] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8050.850591] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8050.881687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8066.401542] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8071.520710] LustreError: 179975:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.142@tcp: failed processing log, type 4: rc = -110 [ 8076.640825] LustreError: 179975:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8076.656526] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8076.665738] LustreError: MGC192.168.204.142@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8076.692137] LustreError: 179975:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8076.712319] LustreError: 179975:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -22 [ 8076.724765] LustreError: 179975:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 8076.834594] Lustre: server umount lustre-OST0000 complete [ 8076.839627] LustreError: 179975:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8085.664648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8087.100082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8091.863863] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8097.250587] LustreError: 180475:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8097.262451] LustreError: 180475:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8100.069864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8100.423152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8104.564481] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8111.949656] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8117.749076] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8124.231839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8124.443590] LustreError: 182290:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8124.451886] Lustre: 182290:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.142@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8124.587472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8130.170942] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8136.529896] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8141.286176] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8141.298060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8141.315924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8143.331979] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8143.336250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8143.353368] Lustre: Skipped 2 previous similar messages [ 8144.884087] Lustre: server umount lustre-MDT0000 complete [ 8148.452729] LustreError: 180474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8148.471146] LustreError: 180474:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8149.234143] LustreError: 180454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788646719 with bad export cookie 1002038751820015674 [ 8149.249270] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8149.257160] LustreError: 180454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8149.556231] Lustre: server umount lustre-MDT0001 complete [ 8163.807493] Lustre: server umount lustre-OST0000 complete [ 8174.538396] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 8186.043279] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 8229.085739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8229.330778] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8229.366590] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8229.475277] Lustre: lustre-MDT0000: new disk, initializing [ 8229.551762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8229.563946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8233.781529] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8245.412213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8245.532744] Lustre: 185817:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8245.616425] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8245.620094] Lustre: Skipped 1 previous similar message [ 8245.689205] Lustre: lustre-MDT0001: new disk, initializing [ 8245.794371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8245.847752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8245.873388] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8250.987851] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8261.940702] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8269.036862] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8277.179097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8277.550248] Lustre: lustre-OST0000: new disk, initializing [ 8277.556665] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8277.568725] Lustre: 187787:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8277.770213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8285.433189] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8287.343808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8287.354075] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8287.522845] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8295.862514] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8300.998381] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8301.223646] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8305.880325] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8306.159854] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8308.193560] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8308.195659] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8308.205623] LustreError: Skipped 1 previous similar message [ 8308.220423] Lustre: Skipped 1 previous similar message [ 8308.225091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8308.241518] Lustre: Skipped 1 previous similar message [ 8313.323360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8314.355517] Lustre: server umount lustre-OST0000 complete [ 8322.529627] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8322.534775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8322.544817] Lustre: Skipped 1 previous similar message [ 8322.551153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8322.562766] Lustre: Skipped 1 previous similar message [ 8325.016806] Lustre: server umount lustre-MDT0000 complete [ 8328.675989] LustreError: 185823:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8328.701725] LustreError: 185823:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8329.158404] LustreError: 185807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788646899 with bad export cookie 1002038751820016395 [ 8329.162849] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8329.172761] LustreError: 185807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8329.530653] Lustre: server umount lustre-MDT0001 complete [ 8340.357622] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 18:21:47 (1788646907) [ 8357.937247] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 8360.393735] Key type lgssc unregistered [ 8360.614432] LNet: 190731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8360.622554] LNetError: 190731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8360.636228] LNet: Removed LNI 192.168.204.142@tcp [ 8361.352904] Key type .llcrypt unregistered [ 8361.355635] Key type ._llcrypt unregistered [ 8377.995790] Key type ._llcrypt registered [ 8378.000452] Key type .llcrypt registered [ 8378.126298] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 8379.171614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8379.354366] alg: No test for adler32 (adler32-zlib) [ 8380.429921] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 8380.716228] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 8382.379731] Key type lgssc registered [ 8383.529563] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8390.257175] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8390.283221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8391.709827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8395.491267] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8401.898189] LustreError: 191948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8401.920880] LustreError: 191948:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8403.875864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8404.304794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8409.039848] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8416.789996] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8423.957564] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8431.247818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8431.496371] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8431.499280] LustreError: 193809:0:(llog.c:1696:llog_backup()) MGC192.168.204.142@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8431.504688] Lustre: 193809:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.142@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8431.611474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8437.218463] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8445.652987] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8449.892170] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8449.903601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8449.909042] Lustre: Skipped 1 previous similar message [ 8449.921967] Lustre: Skipped 1 previous similar message [ 8455.140215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8455.161377] Lustre: Skipped 3 previous similar messages [ 8460.270070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8460.275254] Lustre: Skipped 2 previous similar messages [ 8462.304251] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8462.484166] Lustre: server umount lustre-MDT0000 complete [ 8465.383171] LustreError: 191947:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8465.398896] LustreError: 191947:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8465.814061] LustreError: 192680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788647035 with bad export cookie 6137470758887514237 [ 8465.818393] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8465.821969] LustreError: 192680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8466.296318] LustreError: 192675:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000746775dd x1875532264273280/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8466.495211] Lustre: server umount lustre-MDT0001 complete [ 8479.276848] Lustre: server umount lustre-OST0000 complete [ 8488.615238] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 8499.160047] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 8541.861153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8542.194624] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8542.223729] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8542.317881] Lustre: lustre-MDT0000: new disk, initializing [ 8542.393961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8542.427601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8547.457724] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8558.154189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8558.242398] Lustre: 197337:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8558.291112] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8558.293953] Lustre: Skipped 1 previous similar message [ 8558.297313] LustreError: 198119:0:(llog.c:1696:llog_backup()) MGC192.168.204.142@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8558.303808] LustreError: 198119:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8558.309446] Lustre: 198119:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.142@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8558.326249] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8558.332202] Lustre: Skipped 1 previous similar message [ 8558.432239] Lustre: lustre-MDT0001: new disk, initializing [ 8558.582237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8558.621081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8558.628512] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8562.487611] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8572.201329] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8578.114710] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8585.185193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8585.461412] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8585.463595] Lustre: Skipped 1 previous similar message [ 8585.466541] LustreError: 199281:0:(llog.c:1696:llog_backup()) MGC192.168.204.142@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8585.472731] LustreError: 199281:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8585.478825] Lustre: 199281:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.142@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8585.485914] Lustre: 199281:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8585.505066] Lustre: lustre-OST0000: new disk, initializing [ 8585.508911] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8585.513782] Lustre: 199305:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8585.598623] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8591.726057] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8591.973266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8591.991215] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8592.130551] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8601.665263] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8606.699367] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8606.940867] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8611.787297] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8611.980886] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8617.953584] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8617.961387] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8617.973453] Lustre: Skipped 1 previous similar message [ 8617.977317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8617.982604] Lustre: Skipped 1 previous similar message [ 8623.078247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8628.193985] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8628.324250] Lustre: server umount lustre-OST0000 complete [ 8633.314949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8633.316284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8633.335237] Lustre: Skipped 2 previous similar messages [ 8633.348596] Lustre: Skipped 4 previous similar messages [ 8638.231824] Lustre: server umount lustre-MDT0000 complete [ 8638.447830] LustreError: 197347:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8642.033821] LustreError: 197327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788647211 with bad export cookie 6137470758887514958 [ 8642.042079] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8642.045340] LustreError: 197327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8642.379776] Lustre: server umount lustre-MDT0001 complete [ 8650.619887] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 18:26:58 (1788647218) [ 8667.203743] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 8679.101875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8679.598843] LustreError: 202436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8679.622697] LustreError: 202436:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8679.650838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8685.024854] LustreError: 202437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8686.833153] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8694.248130] LustreError: 202436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8694.267695] LustreError: 202436:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8698.524064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8704.580787] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8707.843551] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8714.998150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8715.405597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8715.420812] Lustre: Skipped 1 previous similar message [ 8721.440584] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8729.052488] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8729.229175] Lustre: lustre-OST0001: new disk, initializing [ 8729.232651] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8729.240532] Lustre: 204746:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8732.364955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8732.375217] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8732.491800] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8735.550782] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8745.962196] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8753.009600] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8755.133752] Lustre: DEBUG MARKER: ADD OST3 [ 8760.884296] loop: module loaded [ 8760.996039] loop0: detected capacity change from 0 to 204800000 [ 8761.176652] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8761.196186] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8761.215503] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8767.135899] loop0: detected capacity change from 0 to 204800000 [ 8768.289898] Lustre: DEBUG MARKER: STOP OST3 [ 8773.179453] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8773.380944] Lustre: lustre-OST0003: new disk, initializing [ 8773.387940] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8773.400960] Lustre: 206733:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8774.860367] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8775.072856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8775.084130] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8775.125954] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8776.864256] LustreError: 206874:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8777.530115] Lustre: Failing over lustre-OST0003 [ 8777.637883] Lustre: server umount lustre-OST0003 complete [ 8780.065255] LustreError: 203924:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8780.281900] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8780.286355] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8780.312973] LustreError: Skipped 1 previous similar message [ 8780.447846] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8790.019458] Lustre: DEBUG MARKER: START OST3 again [ 8797.679971] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8797.992452] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8798.007955] Lustre: Skipped 2 previous similar messages [ 8798.028808] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8799.684668] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8800.047531] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8800.050307] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8805.753427] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8828.505353] Lustre: Failing over lustre-OST0003 [ 8828.599461] Lustre: server umount lustre-OST0003 complete [ 8828.919851] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8828.947636] Lustre: Skipped 1 previous similar message [ 8828.953261] LustreError: 203925:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8828.972640] LustreError: 203925:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 8847.330644] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8847.342967] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8847.352194] Lustre: Skipped 1 previous similar message [ 8847.356485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8852.752888] Lustre: server umount lustre-MDT0000 complete [ 8856.089917] LustreError: 202416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788647426 with bad export cookie 6137470758887515840 [ 8856.115803] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8856.118438] LustreError: 202416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8862.466814] Lustre: server umount lustre-MDT0001 complete [ 8866.834497] Lustre: server umount lustre-OST0000 complete [ 8871.182307] Lustre: server umount lustre-OST0001 complete [ 8878.445967] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 8886.846262] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 8925.393111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8925.708398] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8925.792740] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8925.970345] Lustre: lustre-MDT0000: new disk, initializing [ 8926.032980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8926.064955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8930.829438] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8941.811884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8941.931545] Lustre: 212205:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8941.982945] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8941.986491] Lustre: Skipped 1 previous similar message [ 8942.067603] Lustre: lustre-MDT0001: new disk, initializing [ 8942.201123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8942.225072] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8946.709882] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8956.460916] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8962.352602] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8969.415834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8969.720328] Lustre: 214177:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8971.020754] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8971.176586] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8976.769244] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8986.533907] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8991.673747] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8991.882695] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8996.619199] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8996.925239] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9001.957493] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9001.963720] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9001.964884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9001.964889] Lustre: Skipped 8 previous similar messages [ 9001.975224] Lustre: Skipped 6 previous similar messages [ 9013.218541] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9013.474506] Lustre: server umount lustre-OST0000 complete [ 9018.851971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9023.935699] Lustre: server umount lustre-MDT0000 complete [ 9027.557095] LustreError: 212214:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9027.570042] LustreError: 212214:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 9027.592803] LustreError: 212198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788647597 with bad export cookie 6137470758887518136 [ 9027.598367] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9027.604744] LustreError: 212198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9027.909474] Lustre: server umount lustre-MDT0001 complete [ 9031.326817] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 18:33:18 (1788647598) [ 9045.857549] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 9056.034509] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 9094.267525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9094.505286] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9094.542609] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9094.550733] Lustre: Skipped 1 previous similar message [ 9094.666750] Lustre: lustre-MDT0000: new disk, initializing [ 9094.673432] Lustre: Skipped 1 previous similar message [ 9094.823876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9094.839072] Lustre: Skipped 1 previous similar message [ 9099.645951] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9111.414729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9111.566987] Lustre: 218787:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9111.860834] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9116.859573] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9126.201440] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9131.931767] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9139.146428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9139.364691] Lustre: 220755:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9140.758841] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9145.456734] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9154.475652] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9160.107485] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9160.369472] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9166.169791] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9166.441664] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9171.431062] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9171.432500] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9171.469656] Lustre: Skipped 2 previous similar messages [ 9171.479704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9171.502505] Lustre: Skipped 9 previous similar messages [ 9183.200191] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9183.416442] Lustre: server umount lustre-OST0000 complete [ 9188.837415] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9188.852121] LustreError: Skipped 1 previous similar message [ 9194.494858] Lustre: server umount lustre-MDT0000 complete [ 9197.027137] LustreError: 218798:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9197.051976] LustreError: 218798:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9199.668844] LustreError: 219582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788647769 with bad export cookie 6137470758887519011 [ 9199.679225] LustreError: 219582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9199.679248] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9200.490323] Lustre: server umount lustre-MDT0001 complete [ 9212.021746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9212.759604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9212.767545] Lustre: Skipped 5 previous similar messages [ 9218.368518] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9228.974642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9235.931143] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9246.860845] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9254.013836] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9263.831057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9271.665342] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9280.025096] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9298.404649] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9298.411358] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9298.441312] Lustre: Skipped 4 previous similar messages [ 9302.161286] Lustre: server umount lustre-OST0000 complete [ 9306.596752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9306.603178] Lustre: Skipped 9 previous similar messages [ 9312.732808] Lustre: server umount lustre-MDT0000 complete [ 9317.509310] LustreError: 222959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788647887 with bad export cookie 6137470758887519900 [ 9317.538466] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9317.538975] LustreError: 222959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9318.213773] Lustre: server umount lustre-MDT0001 complete [ 9421.815732] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 9426.123223] Key type lgssc unregistered [ 9426.571704] LNet: 226662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9426.588806] LNetError: 226662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9426.607413] LNet: Removed LNI 192.168.204.142@tcp [ 9428.350234] Key type .llcrypt unregistered [ 9428.354158] Key type ._llcrypt unregistered [ 9465.868609] Key type ._llcrypt registered [ 9465.871629] Key type .llcrypt registered [ 9465.978959] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 9486.693588] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 9488.102548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9488.423540] alg: No test for adler32 (adler32-zlib) [ 9489.591824] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 9489.912561] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 9491.688764] Key type lgssc registered [ 9493.751965] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9540.062839] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9540.098298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9541.433339] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9541.476666] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9541.586886] Lustre: lustre-MDT0000: new disk, initializing [ 9541.770817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9541.806662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9548.771890] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9562.124213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9562.296790] Lustre: 230015:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9562.436827] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9562.441593] Lustre: Skipped 1 previous similar message [ 9562.625939] Lustre: lustre-MDT0001: new disk, initializing [ 9562.804968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9562.873640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9562.895060] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9568.903721] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9579.717354] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9586.681956] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9595.784504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9596.059813] Lustre: lustre-OST0000: new disk, initializing [ 9596.063779] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9596.070645] Lustre: 231985:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9596.197654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9601.689103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9601.715555] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9601.997113] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9605.146110] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9618.942854] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9625.998192] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9626.244973] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9631.847594] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9632.163849] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9637.868610] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9637.874423] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9637.881108] LustreError: Skipped 1 previous similar message [ 9637.891700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9637.922250] Lustre: Skipped 1 previous similar message [ 9640.645902] Lustre: server umount lustre-OST0000 complete [ 9648.098390] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9648.142669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9648.146394] Lustre: Skipped 2 previous similar messages [ 9651.611723] Lustre: server umount lustre-MDT0000 complete [ 9656.269639] LustreError: 231984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788648226 with bad export cookie 8990455942105394534 [ 9656.270825] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9656.285312] LustreError: 231984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9657.102461] Lustre: server umount lustre-MDT0001 complete [ 9680.795910] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 9691.699208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9692.163724] LustreError: 234944:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9692.185468] LustreError: 234944:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9692.211139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9697.252912] LustreError: 234945:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9697.296250] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9702.373092] LustreError: 234944:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9706.501722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9706.953752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9712.079422] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9715.290909] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9722.034996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9722.413634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9728.834942] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9737.578413] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9737.849681] Lustre: lustre-OST0001: new disk, initializing [ 9737.871903] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9737.891973] Lustre: 237252:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9746.450225] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9746.589292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9746.608943] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9746.823859] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9760.018325] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9766.949696] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9782.002186] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 18:45:49 (1788648349) [ 9787.879845] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9787.886834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9787.896437] Lustre: Skipped 4 previous similar messages [ 9787.919437] Lustre: Skipped 3 previous similar messages [ 9790.758616] Lustre: server umount lustre-MDT0000 complete [ 9792.995665] LustreError: 236726:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9793.023525] LustreError: 236726:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9794.699572] LustreError: 234926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788648364 with bad export cookie 8990455942105395416 [ 9794.709179] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9794.717565] LustreError: 234926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9795.024056] Lustre: server umount lustre-MDT0001 complete [ 9809.163953] Lustre: server umount lustre-OST0000 complete [ 9823.185874] Lustre: server umount lustre-OST0001 complete [ 9829.984306] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 9840.513073] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 9876.062254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9876.287218] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9876.316224] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9876.387595] Lustre: lustre-MDT0000: new disk, initializing [ 9876.476231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9876.486789] Lustre: Skipped 1 previous similar message [ 9876.505931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9881.706794] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9891.877108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9891.989814] Lustre: 241200:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9892.043713] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9892.046860] Lustre: Skipped 1 previous similar message [ 9892.110377] Lustre: lustre-MDT0001: new disk, initializing [ 9892.237285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9892.252737] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9896.794872] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9905.470726] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9910.935929] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9918.573936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9918.755899] Lustre: lustre-OST0000: new disk, initializing [ 9918.759218] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9918.764747] Lustre: 243168:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9918.878210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9918.882608] Lustre: Skipped 1 previous similar message [ 9919.893829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9919.912746] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9920.079234] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9926.060213] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9937.032681] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9941.979677] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9942.179580] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9946.793461] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9947.005360] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9950.690958] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9950.694532] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9950.709656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9950.730963] Lustre: Skipped 1 previous similar message [ 9955.141243] Lustre: server umount lustre-OST0000 complete [ 9958.882831] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9958.895503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9958.915648] Lustre: Skipped 1 previous similar message [ 9958.925895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9964.846408] Lustre: server umount lustre-MDT0000 complete [ 9966.051149] LustreError: 241211:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9966.069938] LustreError: 241211:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9968.947572] LustreError: 243167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788648538 with bad export cookie 8990455942105397551 [ 9968.956154] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9968.962323] LustreError: 243167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9969.356366] Lustre: server umount lustre-MDT0001 complete [ 9978.070586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9978.535559] LustreError: 245384:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9983.399209] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9991.662483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9992.105758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9992.116785] Lustre: Skipped 1 previous similar message [ 9997.008448] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10005.693135] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10012.479517] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10020.489373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10028.319191] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10034.894681] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10183.659936] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10183.668250] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10183.680169] Lustre: Skipped 1 previous similar message [10183.686876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10183.691913] Lustre: Skipped 2 previous similar messages [10186.940456] Lustre: server umount lustre-OST0000 complete [10191.844189] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10191.851530] LustreError: Skipped 1 previous similar message [10200.033370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10200.041098] Lustre: Skipped 6 previous similar messages [10205.152115] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10205.328843] Lustre: server umount lustre-MDT0000 complete [10209.249805] LustreError: 245381:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10209.260856] LustreError: 245381:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10209.480597] LustreError: 245367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788648779 with bad export cookie 8990455942105398426 [10209.495397] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10209.851778] Lustre: server umount lustre-MDT0001 complete [10303.642500] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [10306.605936] Key type lgssc unregistered [10306.906593] LNet: 248996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10306.925493] LNetError: 248996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10306.953454] LNet: Removed LNI 192.168.204.142@tcp [10307.787145] Key type .llcrypt unregistered [10307.788879] Key type ._llcrypt unregistered [10333.062180] Key type ._llcrypt registered [10333.065832] Key type .llcrypt registered [10333.185373] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [10352.820704] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [10354.670394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10354.721610] alg: No test for adler32 (adler32-zlib) [10355.843574] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [10356.159486] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [10357.856297] Key type lgssc registered [10359.277563] Lustre: Echo OBD driver; http://www.lustre.org/ [10406.623976] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [10417.517609] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10417.548364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10418.780368] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10418.830678] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10418.934291] Lustre: lustre-MDT0000: new disk, initializing [10419.045415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10419.069692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10423.199493] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10433.443961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10433.593535] Lustre: 253067:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10433.641049] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10433.645499] Lustre: Skipped 1 previous similar message [10433.719759] Lustre: lustre-MDT0001: new disk, initializing [10433.808081] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10433.839660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10433.857898] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10438.172375] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10442.990942] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10450.630300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10450.954560] Lustre: lustre-OST0000: new disk, initializing [10450.961694] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10450.973889] Lustre: 254696:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10451.124772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10452.616574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10452.642527] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10452.839812] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10458.980053] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10471.133102] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10471.405221] Lustre: lustre-OST0001: new disk, initializing [10471.411329] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10471.422606] Lustre: 255569:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10471.537426] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10477.235189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10477.247996] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10477.393835] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10478.196345] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10490.337657] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10497.870714] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10510.820460] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10510.829671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10510.850346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10513.379654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10513.380349] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10515.153771] Lustre: server umount lustre-MDT0000 complete [10518.802425] LustreError: 253860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788649088 with bad export cookie 5977916177443943460 [10518.806982] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10518.815949] LustreError: 253860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10519.227462] Lustre: server umount lustre-MDT0001 complete [10533.967041] Lustre: server umount lustre-OST0000 complete [10548.537496] Lustre: server umount lustre-OST0001 complete [10566.617181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10579.181075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10588.345905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10599.474902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10623.619638] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [10635.121656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10635.140487] Lustre: lustre-MDT0000: reset Object Index mappings [10635.568022] LustreError: 260582:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10635.587038] LustreError: 260582:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10635.654841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10640.709238] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10640.866150] LustreError: 260583:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10645.988438] LustreError: 260582:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10650.015790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10650.029357] Lustre: lustre-MDT0001: reset Object Index mappings [10650.516857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10655.069179] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10657.803822] Lustre: 261724:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10664.243304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10671.059888] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10675.876248] LustreError: 262078:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10679.088067] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10679.465525] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10679.484939] Lustre: Skipped 1 previous similar message [10680.530114] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10680.545784] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10686.202340] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10693.794939] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10697.733701] Lustre: 263607:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10705.550556] Lustre: server umount lustre-MDT0000 complete [10706.422889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10706.450901] Lustre: Skipped 2 previous similar messages [10706.460521] LustreError: 260579:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10706.490110] LustreError: 260579:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10709.935611] LustreError: 260562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788649279 with bad export cookie 5977916177443950250 [10709.950964] LustreError: 260562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10709.954713] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10710.545662] Lustre: server umount lustre-MDT0001 complete [10725.675841] Lustre: server umount lustre-OST0000 complete [10727.520249] Lustre: 250571:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788649281/real 1788649281] req@ffff98450a3e7b80 x1875534336355200/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788649297 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10727.554262] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10727.564708] Lustre: Skipped 1 previous similar message [10729.345901] Lustre: server umount lustre-OST0001 complete [10746.901774] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 19:01:54 (1788649314) [10762.248771] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [10771.620868] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [10811.974920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10812.237734] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10812.261447] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10812.335668] Lustre: lustre-MDT0000: new disk, initializing [10812.419056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10812.437256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10817.343702] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10828.706801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10828.850905] Lustre: 267861:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10828.958650] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10828.964732] Lustre: Skipped 1 previous similar message [10829.088438] Lustre: lustre-MDT0001: new disk, initializing [10829.290112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10829.303118] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10834.572216] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10845.482857] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10852.102203] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10860.363170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10860.692590] Lustre: lustre-OST0000: new disk, initializing [10860.695410] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10860.699750] Lustre: 269832:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10862.245031] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10862.261600] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10862.422491] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10868.496706] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10879.591306] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10884.529333] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10884.723345] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10889.634841] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10889.914917] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10893.282231] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10893.300042] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10893.304216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10893.304229] Lustre: Skipped 3 previous similar messages [10893.308050] Lustre: Skipped 1 previous similar message [10893.351729] LustreError: Skipped 1 previous similar message [10898.356848] Lustre: server umount lustre-OST0000 complete [10903.524218] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10903.543339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10903.555760] Lustre: Skipped 2 previous similar messages [10908.649605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10908.651835] Lustre: Skipped 1 previous similar message [10908.860988] Lustre: server umount lustre-MDT0000 complete [10912.850569] LustreError: 267854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788649482 with bad export cookie 5977916177443952343 [10912.852365] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10912.874300] LustreError: 267854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10913.312435] Lustre: server umount lustre-MDT0001 complete [10937.717714] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [10949.446734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10949.894868] LustreError: 272767:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10949.995401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10950.015856] Lustre: Skipped 2 previous similar messages [10955.469821] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10963.871326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10969.298249] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10972.870843] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10980.917661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10988.297688] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10998.277374] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10998.497832] Lustre: lustre-OST0001: new disk, initializing [10998.502463] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10998.508175] Lustre: 275076:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11000.098134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11000.114172] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11000.298461] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11005.832591] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11015.917247] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11019.689977] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11025.304646] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11025.319307] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [11036.134505] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11036.139355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11036.154235] Lustre: Skipped 3 previous similar messages [11036.158037] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11036.168397] Lustre: Skipped 3 previous similar messages [11037.381627] LustreError: 276623:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffff9845059a8f00) refcount nonzero (2) after lock cleanup; forcing cleanup. [11037.445388] Lustre: server umount lustre-MDT0000 complete [11040.803687] LustreError: 273900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788649610 with bad export cookie 5977916177443953225 [11040.807784] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11040.813642] LustreError: 273900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11041.196970] Lustre: server umount lustre-MDT0001 complete [11056.042826] Lustre: server umount lustre-OST0000 complete [11057.952129] Lustre: 250573:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788649611/real 1788649611] req@ffff9845027cf480 x1875534336459648/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788649627 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11060.393758] Lustre: server umount lustre-OST0001 complete [11079.901118] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [11091.057529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11091.532283] LustreError: 278418:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11091.548232] LustreError: 278418:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11091.609558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11091.620807] Lustre: Skipped 3 previous similar messages [11096.727421] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11106.374121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11112.500941] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11116.081445] Lustre: 279558:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11123.711542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11130.874216] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11131.448894] LustreError: 279913:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11131.474486] LustreError: 279913:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11137.605720] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11141.992624] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11147.782766] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11149.590520] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11157.321541] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11161.856508] Lustre: 281434:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11167.613807] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11167.731631] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11184.731142] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 19:09:11 (1788649751) [11204.077133] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11204.089238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11204.111546] Lustre: Skipped 4 previous similar messages [11204.124076] Lustre: Skipped 1 previous similar message [11204.147473] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11216.352781] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11216.634197] Lustre: server umount lustre-MDT0000 complete [11219.431970] LustreError: 282323:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11219.461938] LustreError: 282323:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [11221.339870] LustreError: 278399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788649791 with bad export cookie 5977916177443959637 [11221.351913] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11228.067282] Lustre: server umount lustre-MDT0001 complete [11238.870346] Lustre: server umount lustre-OST0000 complete [11243.395246] Lustre: server umount lustre-OST0001 complete [11251.652960] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [11263.152427] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [11318.517878] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [11331.854421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11332.275816] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11332.322130] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11332.493170] Lustre: lustre-MDT0000: new disk, initializing [11332.695509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11340.935910] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11362.013497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11362.355776] Lustre: 285974:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11362.892262] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11362.910462] Lustre: Skipped 4 previous similar messages [11363.041440] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11372.366199] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11379.850478] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11388.068775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11388.542455] Lustre: 287609:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11390.309390] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11396.523668] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11409.156936] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11409.400218] Lustre: 288481:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11411.195479] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11411.208035] Lustre: Skipped 1 previous similar message [11411.336347] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11416.367408] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11426.348772] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11448.981179] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 19:13:36 (1788650016) [11452.389465] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11452.398218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11452.411330] Lustre: Skipped 6 previous similar messages [11452.426108] Lustre: Skipped 17 previous similar messages [11453.384917] Lustre: server umount lustre-MDT0000 complete [11457.519630] LustreError: 285987:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11457.543518] LustreError: 285987:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [11457.639241] LustreError: 288480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788650027 with bad export cookie 5977916177443967568 [11457.654416] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11457.667706] LustreError: 288480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [11458.272762] Lustre: server umount lustre-MDT0001 complete [11473.433584] Lustre: server umount lustre-OST0000 complete [11488.876625] Lustre: server umount lustre-OST0001 complete [11503.084524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11509.639318] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11520.398781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11526.947904] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11536.061612] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11544.881900] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11554.513293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11563.351197] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11573.119429] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11592.811096] loop1: detected capacity change from 0 to 204800000 [11593.124967] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11593.143858] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11593.167406] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11599.264162] loop1: detected capacity change from 0 to 204800000 [11606.891416] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11607.017076] LustreError: 290722:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs18211: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11607.035744] LustreError: 290722:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs18211 [11607.048302] LustreError: 290722:0:(mgs_handler.c:517:mgs_target_reg()) fs18211-MDT0001: Error checking target index: rc = -13 [11607.058711] LustreError: fs18211-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11607.069816] LustreError: 294452:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [11607.077453] LustreError: 294452:0:(tgt_mount.c:2132:server_put_super()) no obd fs18211-MDT0001 [11607.085151] LustreError: 294452:0:(tgt_mount.c:116:server_deregister_mount()) fs18211-MDT0001 not registered [11607.113410] Lustre: server umount fs18211-MDT0001 complete [11607.116448] LustreError: 294452:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11617.093073] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11617.202680] Lustre: Setting parameter fs18211-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs18211-MDT0001 [11617.212848] Lustre: Skipped 1 previous similar message [11617.288241] Lustre: srv-fs18211-MDT0001: No data found on store. Initialize space. [11617.295186] Lustre: Skipped 4 previous similar messages [11617.421783] Lustre: fs18211-MDT0001: new disk, initializing [11617.425716] Lustre: Skipped 3 previous similar messages [11617.565569] Lustre: srv-fs18211-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11617.574427] Lustre: 294920:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs18211-MDT0001-mdtlov: can't create new llog: rc = -115 [11617.583466] LustreError: 294920:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs18211-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11623.318779] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11631.293493] Lustre: server umount fs18211-MDT0001 complete [11645.418559] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11645.702921] loop1: detected capacity change from 0 to 204800000 [11652.692806] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11653.134819] Lustre: 296490:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs18211-MDT0001-mdtlov: can't create new llog: rc = -115 [11653.148681] LustreError: 296490:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs18211-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11657.862469] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11663.506626] Lustre: server umount fs18211-MDT0001 complete [11676.134182] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11676.140193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11676.158109] Lustre: Skipped 2 previous similar messages [11676.172788] Lustre: Skipped 2 previous similar messages [11681.527693] Lustre: server umount lustre-MDT0000 complete [11692.346863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11692.446705] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11698.160048] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11698.213652] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11708.791938] loop1: detected capacity change from 0 to 204800000 [11715.433610] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11715.870760] Lustre: 298934:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs18211-MDT0001-mdtlov: can't create new llog: rc = -115 [11715.881470] LustreError: 298934:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs18211-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11717.366808] LustreError: 292566:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11717.388328] LustreError: 292566:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 100 previous similar messages [11721.418256] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11727.683819] Lustre: server umount fs18211-MDT0001 complete [11759.634244] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11759.651326] LustreError: Skipped 1 previous similar message [11769.312515] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11769.530762] Lustre: server umount lustre-OST0000 complete [11780.720599] Lustre: server umount lustre-MDT0000 complete [11784.669172] LustreError: 290714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788650354 with bad export cookie 5977916177443970249 [11784.669841] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11784.676406] LustreError: 290714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11785.206522] Lustre: server umount lustre-MDT0001 complete [11800.211370] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [11803.912127] Key type lgssc unregistered [11804.299434] LNet: 301281:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11804.346817] LNetError: 301281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11805.417891] LNet: Removed LNI 192.168.204.142@tcp [11806.086988] Key type .llcrypt unregistered [11806.093432] Key type ._llcrypt unregistered [11830.395658] Key type ._llcrypt registered [11830.404142] Key type .llcrypt registered [11830.562327] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [11831.402528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11831.428862] alg: No test for adler32 (adler32-zlib) [11832.637854] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [11833.055248] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [11834.824706] Key type lgssc registered [11836.116507] Lustre: Echo OBD driver; http://www.lustre.org/ [11844.584975] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11844.611191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11846.086767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11851.187293] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11855.345355] LustreError: 302552:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11855.366322] LustreError: 302552:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11860.449286] LustreError: 302553:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11860.591726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11860.905449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11865.306272] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11874.485603] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11880.953292] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11888.408499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11888.830555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11890.930864] LustreError: 304383:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11895.280751] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11896.291630] LustreError: 304384:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11901.433241] LustreError: 304382:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11904.619658] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11910.640958] LustreError: 304382:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11910.664996] LustreError: 304382:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11924.911299] loop1: detected capacity change from 0 to 204800000 [11925.129572] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11925.148631] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11925.198429] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11928.312496] LustreError: 304382:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11928.348502] LustreError: 304382:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [11931.461733] loop1: detected capacity change from 0 to 204800000 [11938.495900] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11938.697322] Lustre: fs18500-OST0000: new disk, initializing [11938.702628] Lustre: srv-fs18500-OST0000: No data found on store. Initialize space. [11938.713625] Lustre: 306292:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11938.797025] Lustre: fs18500-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11946.214892] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11955.219162] Lustre: server umount fs18500-OST0000 complete [11961.832435] LustreError: 304384:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11961.856826] LustreError: 304384:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [11968.869301] loop1: detected capacity change from 0 to 204800000 [11972.088382] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11974.842611] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11975.100360] Lustre: fs18500-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11981.748894] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11987.450845] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11991.692894] loop2: detected capacity change from 0 to 204800000 [11992.067111] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11992.083244] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11992.098737] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11999.119700] loop2: detected capacity change from 0 to 204800000 [12005.045756] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12005.120281] LustreError: fs18500-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [12005.130983] LustreError: fs18500-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [12005.138435] LustreError: 309332:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [12005.143742] LustreError: 309332:0:(tgt_mount.c:2132:server_put_super()) no obd fs18500-OST0001 [12005.148249] LustreError: 309332:0:(tgt_mount.c:116:server_deregister_mount()) fs18500-OST0001 not registered [12005.173215] Lustre: server umount fs18500-OST0001 complete [12005.187578] LustreError: 309332:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [12017.423830] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12017.539862] Lustre: fs18500-OST0001: new disk, initializing [12017.543422] Lustre: srv-fs18500-OST0001: No data found on store. Initialize space. [12017.548702] Lustre: 309883:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12017.628111] Lustre: fs18500-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12025.806264] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12028.402883] LustreError: 304382:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12028.423889] LustreError: 304382:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [12036.217472] Lustre: server umount fs18500-OST0000 complete [12046.268561] Lustre: server umount fs18500-OST0001 complete [12059.091796] loop1: detected capacity change from 0 to 204800000 [12065.459574] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12065.634346] Lustre: fs18500-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12072.704903] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12081.007871] loop2: detected capacity change from 0 to 204800000 [12087.463329] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12094.835908] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12105.861454] Lustre: server umount fs18500-OST0000 complete [12116.980948] Lustre: server umount fs18500-OST0001 complete [12123.680024] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12131.809812] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12131.831112] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12131.860990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12133.859816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12133.866268] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12138.329108] Lustre: server umount lustre-OST0000 complete [12142.564216] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12142.575829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12142.601926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12142.612203] Lustre: Skipped 1 previous similar message [12149.000306] Lustre: server umount lustre-MDT0000 complete [12153.824060] LustreError: 304407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788650723 with bad export cookie 1146819155841699926 [12153.826742] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12153.842241] LustreError: 304407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12154.366460] Lustre: server umount lustre-MDT0001 complete [12166.681741] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [12170.002180] Key type lgssc unregistered [12170.322907] LNet: 315806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12170.331955] LNetError: 315806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12170.342847] LNet: Removed LNI 192.168.204.142@tcp [12171.373361] Key type .llcrypt unregistered [12171.376461] Key type ._llcrypt unregistered [12207.364932] Key type ._llcrypt registered [12207.368523] Key type .llcrypt registered [12207.603738] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [12225.127543] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [12227.162977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12227.182776] alg: No test for adler32 (adler32-zlib) [12228.365781] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [12228.634395] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [12230.408184] Key type lgssc registered [12231.537224] Lustre: Echo OBD driver; http://www.lustre.org/ [12272.718468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12272.770478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12274.087823] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12274.163672] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12274.319232] Lustre: lustre-MDT0000: new disk, initializing [12274.428814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12274.461775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12279.404634] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12291.887381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12292.018371] Lustre: 319662:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12292.060277] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12292.065029] Lustre: Skipped 1 previous similar message [12292.119384] Lustre: lustre-MDT0001: new disk, initializing [12292.193598] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12292.230938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12292.240533] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12296.867601] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12306.627764] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12313.216420] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12320.726214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12321.213503] Lustre: lustre-OST0000: new disk, initializing [12321.221201] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12321.236197] Lustre: 321633:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12321.350965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12327.608858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12327.634727] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12327.893877] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12329.971851] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12342.989564] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12345.946810] Lustre: Mounted lustre-client - version 2.17.58_3_g980e1e2 [12347.179141] Lustre: Unmounted lustre-client [12358.626646] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12358.627469] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12358.631237] LustreError: Skipped 1 previous similar message [12358.650747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12358.672937] Lustre: Skipped 1 previous similar message [12361.212081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12361.226341] Lustre: Skipped 1 previous similar message [12363.872500] Lustre: server umount lustre-OST0000 complete [12368.865867] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12368.875510] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12368.885921] Lustre: Skipped 1 previous similar message [12368.900686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12371.436040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12371.436806] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12376.393462] Lustre: server umount lustre-MDT0000 complete [12380.284852] LustreError: 319654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788650950 with bad export cookie 422970418980097139 [12380.286154] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12380.296038] LustreError: 319654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12380.790302] Lustre: server umount lustre-MDT0001 complete [12391.652672] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [12394.483877] Key type lgssc unregistered [12397.600810] LNet: 1 local NIs in recovery (showing 1): 192.168.204.142@tcp [12397.921445] LNet: 323794:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12397.930418] LNetError: 323794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12477.801477] LNet: Removed LNI 192.168.204.142@tcp [12478.551893] Key type .llcrypt unregistered [12478.553926] Key type ._llcrypt unregistered [12497.435793] Key type ._llcrypt registered [12497.439290] Key type .llcrypt registered [12497.559261] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [12514.377921] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [12515.707933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12515.737730] alg: No test for adler32 (adler32-zlib) [12516.818555] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [12517.134718] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [12518.872323] Key type lgssc registered [12520.081859] Lustre: Echo OBD driver; http://www.lustre.org/ [12555.598176] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 19:32:02 (1788651122) [12563.872782] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12563.926747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12565.312264] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12565.366402] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12565.491573] Lustre: lustre-MDT0000: new disk, initializing [12565.669519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12565.694322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12570.868617] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12582.238306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12582.413357] Lustre: 327016:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12582.520648] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12582.529225] Lustre: Skipped 1 previous similar message [12582.657938] Lustre: lustre-MDT0001: new disk, initializing [12582.813432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12582.889709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12582.895610] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12588.137353] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12597.176604] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12603.169986] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12610.516324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12610.726980] Lustre: lustre-OST0000: new disk, initializing [12610.730800] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12610.736563] Lustre: 328985:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12610.834630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12617.176486] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12619.322210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12619.338436] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12619.451054] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12627.247610] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12645.349680] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12645.360671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12645.373839] Lustre: Skipped 1 previous similar message [12645.375249] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12645.381637] Lustre: Skipped 1 previous similar message [12647.981350] Lustre: server umount lustre-OST0000 complete [12654.565354] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12654.578728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12654.592178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12655.591867] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12655.596068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12655.615947] Lustre: Skipped 1 previous similar message [12660.707282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12660.713029] Lustre: Skipped 1 previous similar message [12665.829456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12665.841267] Lustre: Skipped 1 previous similar message [12666.336501] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12666.595691] Lustre: server umount lustre-MDT0000 complete [12671.006285] LustreError: 327008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788651240 with bad export cookie 1159160530038199319 [12671.014756] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12671.026022] LustreError: 327008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12671.370275] Lustre: server umount lustre-MDT0001 complete [12684.290963] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [12686.933398] Key type lgssc unregistered [12687.235646] LNet: 330893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12687.245496] LNetError: 330893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12687.271326] LNet: Removed LNI 192.168.204.142@tcp [12688.129886] Key type .llcrypt unregistered [12688.132704] Key type ._llcrypt unregistered [12713.602169] Key type ._llcrypt registered [12713.608620] Key type .llcrypt registered [12713.726701] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [12714.632044] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12714.733531] alg: No test for adler32 (adler32-zlib) [12715.842391] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [12716.180739] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [12717.856213] Key type lgssc registered [12719.056743] Lustre: Echo OBD driver; http://www.lustre.org/ [12727.891368] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12727.925984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12729.383926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12734.058650] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12739.561934] LustreError: 332169:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12739.585712] LustreError: 332169:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12743.662694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12744.178222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12749.531115] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12758.385498] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12766.001332] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12774.253752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12774.663622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12782.613925] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12792.068356] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12803.289849] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12805.556675] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12810.728191] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12810.730790] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12810.734108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12810.791528] Lustre: Skipped 2 previous similar messages [12814.598575] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.42@tcp (stopping) [12814.607401] Lustre: Skipped 2 previous similar messages [12815.848986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12815.865992] Lustre: Skipped 2 previous similar messages [12816.112758] Lustre: server umount lustre-MDT0000 complete [12819.702953] LustreError: 332165:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12819.919852] LustreError: 332150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788651389 with bad export cookie 15841337283246287463 [12819.946946] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12820.961434] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12820.962213] LustreError: 333457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12820.978276] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12824.817609] LustreError: 332164:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12824.995462] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12825.010465] Lustre: Skipped 1 previous similar message [12829.091518] LustreError: 333458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12829.114537] LustreError: 333458:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12834.272607] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12834.274530] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12834.306565] Lustre: Skipped 3 previous similar messages [12834.665940] Lustre: server umount lustre-MDT0001 complete [12838.888523] Lustre: server umount lustre-OST0000 complete [12848.134097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12854.125675] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12857.337841] Lustre: 336356:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.142@tcp [12860.137252] Lustre: server umount MGS complete [12868.897594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12869.240698] LustreError: 336927:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12869.251184] LustreError: 336927:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12869.305802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12874.590914] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12884.148585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12884.589125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12890.065623] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12900.317385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12900.798493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12907.227944] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12915.276509] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12917.242413] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12923.106735] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12925.033082] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12934.625344] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12934.634400] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12934.658486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12934.667828] Lustre: Skipped 1 previous similar message [12934.676275] Lustre: Skipped 1 previous similar message [12937.872732] Lustre: server umount lustre-OST0000 complete [12944.871518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12944.902836] Lustre: Skipped 1 previous similar message [12949.012644] Lustre: server umount lustre-MDT0000 complete [12949.987805] LustreError: 337657:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12950.016446] LustreError: 337657:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12953.684578] LustreError: 336907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788651523 with bad export cookie 15841337283246288303 [12953.692480] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12953.714126] LustreError: 336907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [12954.295744] Lustre: server umount lustre-MDT0001 complete [12966.335876] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [12969.944612] Key type lgssc unregistered [12970.307554] LNet: 340363:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12970.318375] LNetError: 340363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12970.343802] LNet: Removed LNI 192.168.204.142@tcp [12971.460566] Key type .llcrypt unregistered [12971.462915] Key type ._llcrypt unregistered [12995.927091] Key type ._llcrypt registered [12995.942149] Key type .llcrypt registered [12996.139400] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [13012.706712] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [13014.780758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13014.823380] alg: No test for adler32 (adler32-zlib) [13016.083940] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [13016.466111] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [13018.216243] Key type lgssc registered [13020.221367] Lustre: Echo OBD driver; http://www.lustre.org/ [13068.521600] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13068.579463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13069.899314] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13069.942668] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13070.069257] Lustre: lustre-MDT0000: new disk, initializing [13070.226401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13070.244028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13076.420564] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [13087.581584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13087.746391] Lustre: 343524:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13087.824399] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13087.828917] Lustre: Skipped 1 previous similar message [13087.943136] Lustre: lustre-MDT0001: new disk, initializing [13088.067288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13088.146867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13088.156650] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13093.236910] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [13103.964736] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13110.734914] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13118.086119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13118.504227] Lustre: lustre-OST0000: new disk, initializing [13118.508450] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13118.517829] Lustre: 345492:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13118.643794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13124.680757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13124.690959] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13124.872650] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13126.482276] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [13139.094263] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13145.930690] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13146.185280] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13152.557404] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13152.843387] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13155.811790] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13155.816889] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13155.833220] LustreError: Skipped 1 previous similar message [13155.839976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13155.851915] Lustre: Skipped 1 previous similar message [13160.931288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13160.952668] Lustre: Skipped 2 previous similar messages [13161.611839] Lustre: server umount lustre-OST0000 complete [13170.156083] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13170.163544] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13170.176418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13171.182734] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13172.786753] Lustre: server umount lustre-MDT0000 complete [13176.904830] LustreError: 343516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788651746 with bad export cookie 13037241688133109757 [13176.906393] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13176.916448] LustreError: 343516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13177.273857] Lustre: server umount lustre-MDT0001 complete [13187.995513] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 19:42:35 (1788651755) [13206.150100] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [13217.948680] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [13257.168539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13257.493087] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13257.586970] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13257.693241] Lustre: lustre-MDT0000: new disk, initializing [13257.883562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13257.922594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13262.618741] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [13273.698703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13273.893168] Lustre: 350256:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13273.967525] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13273.978201] Lustre: Skipped 1 previous similar message [13274.071622] Lustre: lustre-MDT0001: new disk, initializing [13274.140589] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13274.180704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13274.195079] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13279.526119] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [13291.572614] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13297.981747] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13306.621392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13307.127215] Lustre: lustre-OST0000: new disk, initializing [13307.129603] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13307.137877] Lustre: 352222:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13307.231716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13313.381645] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [13315.706306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13315.735732] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13315.817926] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13324.609756] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13367.267072] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13367.274120] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13367.272498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13367.274570] Lustre: Skipped 2 previous similar messages [13367.289735] Lustre: Skipped 1 previous similar message [13371.766800] Lustre: server umount lustre-OST0000 complete [13376.482602] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13376.492441] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13376.508617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13376.517429] Lustre: Skipped 1 previous similar message [13378.540397] Lustre: server umount lustre-MDT0000 complete [13382.499791] LustreError: 350248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788651952 with bad export cookie 13037241688133110632 [13382.504254] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13382.516126] LustreError: 350248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13382.982201] Lustre: server umount lustre-MDT0001 complete [13396.842696] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [13399.949548] Key type lgssc unregistered [13400.313041] LNet: 354394:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13400.318694] LNetError: 354394:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13400.338693] LNet: Removed LNI 192.168.204.142@tcp [13401.269701] Key type .llcrypt unregistered [13401.271530] Key type ._llcrypt unregistered [13509.524446] Key type ._llcrypt registered [13509.526583] Key type .llcrypt registered [13509.621747] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [13509.995285] Key type .llcrypt unregistered [13510.001471] Key type ._llcrypt unregistered [13594.451306] Key type ._llcrypt registered [13594.456834] Key type .llcrypt registered [13594.626305] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [13595.132299] Key type .llcrypt unregistered [13595.135103] Key type ._llcrypt unregistered [13630.338396] Key type ._llcrypt registered [13630.343384] Key type .llcrypt registered [13630.542611] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [13631.023382] Key type .llcrypt unregistered [13631.027714] Key type ._llcrypt unregistered [13663.870464] Key type ._llcrypt registered [13663.876508] Key type .llcrypt registered [13663.968469] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [13664.240571] Key type .llcrypt unregistered [13664.248227] Key type ._llcrypt unregistered [13741.370039] Key type ._llcrypt registered [13741.374070] Key type .llcrypt registered [13741.512740] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [13741.792404] Key type .llcrypt unregistered [13741.794817] Key type ._llcrypt unregistered [13776.609474] Key type ._llcrypt registered [13776.615130] Key type .llcrypt registered [13776.825981] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [13777.112389] Key type .llcrypt unregistered [13777.129643] Key type ._llcrypt unregistered [13811.758594] Key type ._llcrypt registered [13811.761990] Key type .llcrypt registered [13811.883551] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [13812.175691] Key type .llcrypt unregistered [13812.178963] Key type ._llcrypt unregistered [13847.815469] Key type ._llcrypt registered [13847.817745] Key type .llcrypt registered [13847.941371] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [13848.204091] Key type .llcrypt unregistered [13848.208387] Key type ._llcrypt unregistered [13883.249867] Key type ._llcrypt registered [13883.254415] Key type .llcrypt registered [13883.342436] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [13883.635402] Key type .llcrypt unregistered [13883.650817] Key type ._llcrypt unregistered [13916.976653] Key type ._llcrypt registered [13916.979798] Key type .llcrypt registered [13917.087740] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [13917.429893] Key type .llcrypt unregistered [13917.439907] Key type ._llcrypt unregistered [13990.848236] Key type ._llcrypt registered [13990.852033] Key type .llcrypt registered [13990.972861] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [13991.243959] Key type .llcrypt unregistered [13991.248181] Key type ._llcrypt unregistered [14026.983701] Key type ._llcrypt registered [14026.996229] Key type .llcrypt registered [14027.175410] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [14027.589454] Key type .llcrypt unregistered [14027.592338] Key type ._llcrypt unregistered [14063.906710] Key type ._llcrypt registered [14063.910172] Key type .llcrypt registered [14064.038283] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [14064.478537] Key type .llcrypt unregistered [14064.483392] Key type ._llcrypt unregistered [14129.293431] Key type ._llcrypt registered [14129.298216] Key type .llcrypt registered [14129.409621] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [14146.885921] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [14148.682633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14148.727117] alg: No test for adler32 (adler32-zlib) [14149.805751] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [14150.208960] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [14151.915984] Key type lgssc registered [14153.302623] Lustre: Echo OBD driver; http://www.lustre.org/ [14193.684579] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14193.716842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14194.975715] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14195.028169] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14195.141945] Lustre: tst250fs-MDT0000: new disk, initializing [14195.335698] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14195.384247] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14201.864937] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [14213.471627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14213.679064] Lustre: 374706:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14213.752520] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14213.756645] Lustre: Skipped 1 previous similar message [14213.854920] Lustre: tst250fs-MDT0001: new disk, initializing [14213.986129] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14214.064344] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14214.088307] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14219.308069] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [14224.843920] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14232.716934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14233.031825] Lustre: tst250fs-OST0000: new disk, initializing [14233.037549] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14233.052290] Lustre: 376337:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14233.195521] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14237.800906] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14237.818192] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14237.942652] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14240.433553] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [14253.842975] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14254.146180] Lustre: tst250fs-OST0001: new disk, initializing [14254.155887] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14254.172442] Lustre: 377211:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14254.340302] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14260.871487] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14260.886503] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14260.997333] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14261.404235] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [14267.508945] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14270.669757] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [14270.678617] Lustre: Skipped 2 previous similar messages [14290.914253] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14290.937779] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14290.976341] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14291.944701] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14291.946224] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14291.992234] Lustre: Skipped 3 previous similar messages [14296.468623] Lustre: server umount tst250fs-MDT0000 complete [14300.601170] LustreError: 374698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788652870 with bad export cookie 7193258296652595041 [14300.617986] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14300.625030] LustreError: 374698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14301.116521] Lustre: server umount tst250fs-MDT0001 complete [14315.665689] Lustre: server umount tst250fs-OST0000 complete [14318.562372] Lustre: 372930:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788652872/real 1788652872] req@ffff9845037f8a80 x1875538314600576/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788652888 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14318.582563] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14318.596375] Lustre: Skipped 2 previous similar messages [14320.161186] Lustre: server umount tst250fs-OST0001 complete [14341.184875] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [14352.314674] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [14392.741122] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 20:02:39 (1788652959) [14395.217650] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14416.462909] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [14430.245562] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [14468.503310] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14163 sec ========= 20:03:55 (1788653035) [14470.383876] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 20:03:57 (1788653037) === [14474.029745] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 20:04:01 (1788653041) === [14522.215952] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [14525.483938] Key type lgssc unregistered [14525.793067] LNet: 386373:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14525.797899] LNetError: 386373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14525.810543] LNet: Removed LNI 192.168.204.142@tcp [14526.410166] Key type .llcrypt unregistered [14526.412711] Key type ._llcrypt unregistered