[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 453302415 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002430] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004017] kvm-guest: setup PV IPIs [ 0.007505] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010144] LSM: Security Framework initializing [ 0.011056] Yama: becoming mindful. [ 0.013036] SELinux: Initializing. [ 0.014063] *** VALIDATE selinux *** [ 0.022273] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026470] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027118] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028122] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030069] *** VALIDATE tmpfs *** [ 0.031385] *** VALIDATE proc *** [ 0.033254] *** VALIDATE cgroup *** [ 0.034012] *** VALIDATE cgroup2 *** [ 0.036128] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038029] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040030] Spectre V2 : User space: Vulnerable [ 0.041007] Speculative Store Bypass: Vulnerable [ 0.043862] debug: unmapping init [mem 0xffffffff87c59000-0xffffffff87c60fff] [ 0.046136] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047640] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048021] ... version: 2 [ 0.049011] ... bit width: 48 [ 0.050011] ... generic registers: 4 [ 0.051013] ... value mask: 0000ffffffffffff [ 0.052013] ... max period: 00007fffffffffff [ 0.053016] ... fixed-purpose events: 3 [ 0.054015] ... event mask: 000000070000000f [ 0.055293] rcu: Hierarchical SRCU implementation. [ 0.057250] smp: Bringing up secondary CPUs ... [ 0.058553] x86: Booting SMP configuration: [ 0.059033] .... node #0, CPUs: #1 #2 #3 [ 0.062136] smp: Brought up 1 node, 4 CPUs [ 0.064013] smpboot: Max logical packages: 1 [ 0.065017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.135923] node 0 deferred pages initialised in 69ms [ 0.139095] devtmpfs: initialized [ 0.140254] x86/mm: Memory block size: 128MB [ 0.143771] gcov: version magic: 0x41383552 [ 0.146270] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.147053] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.149296] pinctrl core: initialized pinctrl subsystem [ 0.150119] [ 0.150672] ************************************************************* [ 0.153014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.156012] ** ** [ 0.158008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.160008] ** ** [ 0.162012] ** This means that this kernel is built to expose internal ** [ 0.164010] ** IOMMU data structures, which may compromise security on ** [ 0.167013] ** your system. ** [ 0.170014] ** ** [ 0.173014] ** If you see this message and you are not debugging the ** [ 0.175010] ** kernel, report this immediately to your vendor! ** [ 0.177024] ** ** [ 0.179010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.181010] ************************************************************* [ 0.184635] NET: Registered protocol family 16 [ 0.186406] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.189061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.191045] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.194033] cpuidle: using governor menu [ 0.195841] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.196467] PCI: Using configuration type 1 for base access [ 0.197138] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.205107] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.207030] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.211073] cryptd: max_cpu_qlen set to 1000 [ 0.214132] ACPI: Added _OSI(Module Device) [ 0.216022] ACPI: Added _OSI(Processor Device) [ 0.217010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.219011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.223903] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.229173] ACPI: Interpreter enabled [ 0.231089] ACPI: PM: (supports S0 S3 S4 S5) [ 0.233040] ACPI: Using IOAPIC for interrupt routing [ 0.235132] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.238443] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.247348] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.250042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.252018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.256086] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.259464] acpiphp: Slot [2] registered [ 0.261193] acpiphp: Slot [5] registered [ 0.263123] acpiphp: Slot [6] registered [ 0.265096] acpiphp: Slot [7] registered [ 0.266125] acpiphp: Slot [8] registered [ 0.268262] acpiphp: Slot [9] registered [ 0.269166] acpiphp: Slot [10] registered [ 0.271178] acpiphp: Slot [3] registered [ 0.272128] acpiphp: Slot [4] registered [ 0.274162] acpiphp: Slot [11] registered [ 0.276153] acpiphp: Slot [12] registered [ 0.278122] acpiphp: Slot [13] registered [ 0.279217] acpiphp: Slot [14] registered [ 0.281151] acpiphp: Slot [15] registered [ 0.283150] acpiphp: Slot [16] registered [ 0.284177] acpiphp: Slot [17] registered [ 0.286145] acpiphp: Slot [18] registered [ 0.287119] acpiphp: Slot [19] registered [ 0.289150] acpiphp: Slot [20] registered [ 0.290155] acpiphp: Slot [21] registered [ 0.292216] acpiphp: Slot [22] registered [ 0.294176] acpiphp: Slot [23] registered [ 0.296127] acpiphp: Slot [24] registered [ 0.298216] acpiphp: Slot [25] registered [ 0.299112] acpiphp: Slot [26] registered [ 0.301124] acpiphp: Slot [27] registered [ 0.303123] acpiphp: Slot [28] registered [ 0.304111] acpiphp: Slot [29] registered [ 0.306111] acpiphp: Slot [30] registered [ 0.308104] acpiphp: Slot [31] registered [ 0.310062] PCI host bridge to bus 0000:00 [ 0.311034] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.313021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.315022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.318022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.321035] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.324043] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.326155] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.329635] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.333429] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.342744] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.348670] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.351028] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.353023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.355029] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.358676] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.361768] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.365047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.367978] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.373015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.385016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.389016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.395519] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.407015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.418015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.442018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.453246] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.461015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.466014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.485017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.497014] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.503015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.511016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.530016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.543889] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.549015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.556014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.570031] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.584263] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.590017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.597016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.613015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.628740] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.636016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.642017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.658015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.669474] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.672371] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.674311] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.677459] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.679316] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.684189] iommu: Default domain type: Passthrough [ 0.686423] SCSI subsystem initialized [ 0.688123] ACPI: bus type USB registered [ 0.690117] usbcore: registered new interface driver usbfs [ 0.692075] usbcore: registered new interface driver hub [ 0.693075] usbcore: registered new device driver usb [ 0.695193] pps_core: LinuxPPS API ver. 1 registered [ 0.698015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.702066] PTP clock support registered [ 0.704308] EDAC MC: Ver: 3.0.0 [ 0.707262] PCI: Using ACPI for IRQ routing [ 0.710035] NetLabel: Initializing [ 0.711007] NetLabel: domain hash size = 128 [ 0.712022] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.714077] NetLabel: unlabeled traffic allowed by default [ 0.716133] vgaarb: loaded [ 0.717247] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.721018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.725000] clocksource: Switched to clocksource kvm-clock [ 0.845427] VFS: Disk quotas dquot_6.6.0 [ 0.847155] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.852031] *** VALIDATE ramfs *** [ 0.854953] *** VALIDATE hugetlbfs *** [ 0.857888] pnp: PnP ACPI init [ 0.861596] pnp: PnP ACPI: found 6 devices [ 0.881380] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.885779] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.888177] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.890854] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.893690] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.896686] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.900093] NET: Registered protocol family 2 [ 0.902978] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.908466] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.912603] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.917891] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.921949] TCP: Hash tables configured (established 65536 bind 65536) [ 0.925194] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.928523] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.931951] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.935529] NET: Registered protocol family 1 [ 0.938112] RPC: Registered named UNIX socket transport module. [ 0.940376] RPC: Registered udp transport module. [ 0.942188] RPC: Registered tcp transport module. [ 0.944176] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.946295] NET: Registered protocol family 44 [ 0.948162] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.950598] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.952590] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.954842] PCI: CLS 0 bytes, default 64 [ 0.957510] Unpacking initramfs... [ 3.018949] debug: unmapping init [mem 0xffff99353cc54000-0xffff99353ffbffff] [ 3.023677] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.026452] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.030081] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.550699] Initialise system trusted keyrings [ 3.554043] Key type blacklist registered [ 3.556433] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.565717] zbud: loaded [ 3.568971] *** VALIDATE nfs *** [ 3.571106] *** VALIDATE nfs4 *** [ 3.573035] pstore: using deflate compression [ 3.576901] Platform Keyring initialized [ 3.687496] NET: Registered protocol family 38 [ 3.689576] Key type asymmetric registered [ 3.691641] Asymmetric key parser 'x509' registered [ 3.694240] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.698162] io scheduler mq-deadline registered [ 3.700287] io scheduler kyber registered [ 3.702199] io scheduler bfq registered [ 3.704790] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.708407] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.711675] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.715126] ACPI: Power Button [PWRF] [ 3.720733] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.728554] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.744589] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.753172] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.770630] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.799682] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.829742] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.834913] Non-volatile memory driver v1.3 [ 3.836660] Linux agpgart interface v0.103 [ 3.874696] virtio_blk virtio1: [vda] 146960 512-byte logical blocks (75.2 MB/71.8 MiB) [ 3.879701] vda: detected capacity change from 0 to 75243520 [ 3.893612] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.896648] vdb: detected capacity change from 0 to 1073741824 [ 3.913894] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.918049] vdc: detected capacity change from 0 to 2621440000 [ 3.936604] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.939973] vdd: detected capacity change from 0 to 2621440000 [ 3.956201] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.959312] vde: detected capacity change from 0 to 4294967296 [ 3.978515] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.981683] vdf: detected capacity change from 0 to 4294967296 [ 3.990325] libphy: Fixed MDIO Bus: probed [ 3.996643] usbcore: registered new interface driver usbserial_generic [ 3.999730] usbserial: USB Serial support registered for generic [ 4.002347] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.006908] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.009087] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.011929] mousedev: PS/2 mouse device common for all mice [ 4.015206] rtc_cmos 00:05: RTC can wake from S4 [ 4.020176] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.022688] rtc_cmos 00:05: registered as rtc0 [ 4.026947] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.027804] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.029864] intel_pstate: CPU model not supported [ 4.035090] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.040550] hid: raw HID events driver (C) Jiri Kosina [ 4.042543] usbcore: registered new interface driver usbhid [ 4.044371] usbhid: USB HID core driver [ 4.045829] drop_monitor: Initializing network drop monitor service [ 4.048012] Initializing XFRM netlink socket [ 4.049892] NET: Registered protocol family 10 [ 4.052744] Segment Routing with IPv6 [ 4.054083] NET: Registered protocol family 17 [ 4.055961] mpls_gso: MPLS GSO support [ 4.062473] RAS: Correctable Errors collector initialized. [ 4.064511] AVX version of gcm_enc/dec engaged. [ 4.065949] AES CTR mode by8 optimization enabled [ 4.138916] sched_clock: Marking stable (4138893499, 0)->(5059842778, -920949279) [ 4.142710] registered taskstats version 1 [ 4.145949] Loading compiled-in X.509 certificates [ 4.148622] zswap: loaded using pool lzo/zbud [ 4.172555] Key type big_key registered [ 4.186226] Key type encrypted registered [ 4.187866] ima: No TPM chip found, activating TPM-bypass! [ 4.189919] ima: Allocated hash algorithm: sha1 [ 4.191806] ima: No architecture policies found [ 4.193698] evm: Initialising EVM extended attributes: [ 4.196174] evm: security.selinux [ 4.198068] evm: security.ima [ 4.199668] evm: security.capability [ 4.201165] evm: HMAC attrs: 0x1 [ 4.203705] rtc_cmos 00:05: setting system clock to 2026-08-28 02:19:35 UTC (1787883575) [ 4.210336] debug: unmapping init [mem 0xffffffff88c03000-0xffffffff88dfffff] [ 4.213642] debug: unmapping init [mem 0xffffffff87982000-0xffffffff87c58fff] [ 4.222118] Write protecting the kernel read-only data: 28672k [ 4.226225] debug: unmapping init [mem 0xffffffff86003000-0xffffffff861fffff] [ 4.230294] debug: unmapping init [mem 0xffffffff86914000-0xffffffff869fffff] [ 4.276783] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.286389] systemd[1]: Detected virtualization kvm. [ 4.288752] systemd[1]: Detected architecture x86-64. [ 4.291420] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.320155] systemd[1]: No hostname configured. [ 4.322175] systemd[1]: Set hostname to . [ 4.324535] random: systemd: uninitialized urandom read (16 bytes read) [ 4.327215] systemd[1]: Initializing machine ID from random generator. [ 4.479787] random: systemd: uninitialized urandom read (16 bytes read) [ 4.485268] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.490634] random: systemd: uninitialized urandom read (16 bytes read) [ 4.502610] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. [ 4.520429] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.110808] device-mapper: uevent: version 1.0.3 [ 5.113490] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ 5.735583] random: fast init done [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 5.903839] virtio_net virtio0 ens2: renamed from eth0 [ 5.911059] scsi host0: ata_piix [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.969616] scsi host1: ata_piix [ 5.971126] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.973332] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.458985] dracut-initqueue[590]: RTNETLINK answers: File exists [ 10.708673] random: crng init done [ 10.710336] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.254785] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.556563] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.809143] SELinux: Disabled at runtime. [ 12.871967] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.882699] systemd[1]: Detected virtualization kvm. [ 12.885136] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.371186] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.374890] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.379712] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.383438] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.386797] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.395608] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.401337] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... [ 13.487991] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Local Encrypted Volumes. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.938501] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.306871] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.417605] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.468056] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.486945] EDAC sbridge: Ver: 1.1.2 [ 16.206734] Key type dns_resolver registered [ 16.515215] NFS: Registering the id_resolver key type [ 16.517917] Key type id_resolver registered [ 16.520037] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg344-server login: [ 39.988213] hrtimer: interrupt took 5789860 ns [ 53.115676] libcfs: loading out-of-tree module taints kernel. [ 53.290556] Key type ._llcrypt registered [ 53.299211] Key type .llcrypt registered [ 53.476725] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 77.608836] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 79.020169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 79.040024] alg: No test for adler32 (adler32-zlib) [ 80.629177] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 81.490067] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 83.303276] Key type lgssc registered [ 85.219271] Lustre: Echo OBD driver; http://www.lustre.org/ [ 105.354432] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 149.014826] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 161.680374] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 161.697872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 163.013364] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 163.042401] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 163.142149] Lustre: lustre-MDT0000: new disk, initializing [ 163.205861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 163.218365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 168.204358] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 182.757359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 182.932687] Lustre: 6521:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 182.984640] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 182.990273] Lustre: Skipped 1 previous similar message [ 183.106304] Lustre: lustre-MDT0001: new disk, initializing [ 183.193062] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 183.219720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 183.232408] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 188.785864] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 196.169760] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 212.173462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 212.820425] Lustre: lustre-OST0000: new disk, initializing [ 212.828266] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 212.862520] Lustre: 8458:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 213.013278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 219.665155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 219.677232] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 219.771404] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 222.360874] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 238.295488] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 238.445986] Lustre: lustre-OST0001: new disk, initializing [ 238.448789] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 238.454027] Lustre: 9532:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 238.504348] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 244.513344] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 247.953553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 247.970659] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 248.057045] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 260.287983] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 269.878213] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 276.965428] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing check_logdir /tmp/testlogs/ [ 283.253373] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing yml_node [ 288.457932] Lustre: DEBUG MARKER: Client: 2.17.57.99 [ 291.133761] Lustre: DEBUG MARKER: MDS: 2.17.57.99 [ 293.607812] Lustre: DEBUG MARKER: OSS: 2.17.57.99 [ 295.472158] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 22:24:24 EDT 2026 [ 313.579879] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 315.214899] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 331.747728] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 331.756853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 331.776128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 335.331554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 335.335624] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 335.337853] Lustre: Skipped 1 previous similar message [ 335.382256] Lustre: Skipped 1 previous similar message [ 337.878255] Lustre: server umount lustre-MDT0000 complete [ 342.388106] LustreError: 8457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787883913 with bad export cookie 462762643850102241 [ 342.399565] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 342.403863] LustreError: 8457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 345.567864] LustreError: 6528:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 345.572175] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 345.583961] LustreError: 6528:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 345.625373] Lustre: Skipped 1 previous similar message [ 345.630634] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 345.635587] Lustre: Skipped 3 previous similar messages [ 347.616811] LustreError: 6527:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 347.638523] LustreError: 6527:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 348.843841] Lustre: server umount lustre-MDT0001 complete [ 352.880823] Lustre: server umount lustre-OST0000 complete [ 356.803875] Lustre: server umount lustre-OST0001 complete [ 362.418495] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 372.428194] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 412.324707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 412.665232] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 412.691860] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 412.803811] Lustre: lustre-MDT0000: new disk, initializing [ 412.929643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 412.964354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 417.616305] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 429.316106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 429.516178] Lustre: 15090:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 429.691501] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 429.696902] Lustre: Skipped 1 previous similar message [ 429.813393] Lustre: lustre-MDT0001: new disk, initializing [ 430.008151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 430.060587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 430.078221] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 435.643385] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 446.816763] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 454.328749] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 462.504436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 462.935419] Lustre: lustre-OST0000: new disk, initializing [ 462.941418] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 462.951526] Lustre: 17061:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 463.099885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 464.700808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 464.723236] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 464.929948] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 470.445566] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 481.213583] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 486.611819] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 486.831679] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 491.829870] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 492.020738] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 495.584776] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 495.599744] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 495.645489] Lustre: Skipped 1 previous similar message [ 495.655187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 495.663490] Lustre: Skipped 2 previous similar messages [ 500.706905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 500.717730] Lustre: Skipped 1 previous similar message [ 508.383146] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 508.775255] Lustre: server umount lustre-OST0000 complete [ 516.065549] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 516.074056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 516.087611] Lustre: Skipped 2 previous similar messages [ 516.103069] Lustre: Skipped 3 previous similar messages [ 519.817945] Lustre: server umount lustre-MDT0000 complete [ 521.188565] LustreError: 15101:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 521.204654] LustreError: 15101:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 523.750847] LustreError: 15082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787884095 with bad export cookie 462762643850103774 [ 523.753226] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 523.762621] LustreError: 15082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 524.128474] Lustre: server umount lustre-MDT0001 complete [ 530.231225] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 22:28:19 (1787884099) [ 550.030730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 550.774905] LustreError: 19937:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 550.840048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 555.546377] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 556.001228] LustreError: 19938:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 564.775914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 569.987032] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 577.097773] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 583.488482] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 585.765852] Lustre: Failing over lustre-MDT0000 [ 585.966746] Lustre: server umount lustre-MDT0000 complete [ 586.722375] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 586.727111] LustreError: 19937:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 586.737498] Lustre: Skipped 1 previous similar message [ 586.753652] LustreError: 19937:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 603.039191] Lustre: 3652:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787884158/real 1787884158] req@ffff9935bdd9c780 x1874731888448384/t0(0) o400->MGC192.168.203.144@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787884174 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 603.088079] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 603.106367] LustreError: 19938:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 603.137488] LustreError: 19938:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 606.194045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 613.344551] LustreError: 3650:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9935bd926580 x1874731888451328/t0(0) o250->MGC192.168.203.144@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 613.803923] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 618.988617] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 618.992116] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 619.026625] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 619.049482] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 630.273172] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 634.411921] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 634.424982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 640.802574] Lustre: server umount lustre-MDT0000 complete [ 640.997896] LustreError: 19933:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 641.021575] LustreError: 19933:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 644.954732] LustreError: 19919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787884216 with bad export cookie 462762643850105188 [ 644.961479] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 644.973880] LustreError: 19919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 645.592683] Lustre: server umount lustre-MDT0001 complete [ 654.477278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 655.163505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 655.174103] Lustre: Skipped 2 previous similar messages [ 659.493995] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 668.042243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 672.828280] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 679.870310] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 685.271084] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 687.334331] Lustre: Failing over lustre-MDT0000 [ 687.674868] Lustre: server umount lustre-MDT0000 complete [ 689.126173] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 689.137779] LustreError: Skipped 1 previous similar message [ 689.142817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 689.159043] Lustre: Skipped 1 previous similar message [ 706.527390] Lustre: 3652:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787884261/real 1787884261] req@ffff99358a875a40 x1874731888478848/t0(0) o400->MGC192.168.203.144@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787884277 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 706.556516] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 706.573204] LustreError: 23271:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 706.598225] LustreError: 23271:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 13 previous similar messages [ 707.371817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 716.008605] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 721.381806] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 721.388278] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 721.390532] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 721.390540] Lustre: Skipped 1 previous similar message [ 721.448287] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 732.231702] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 736.743080] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 736.745432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 736.775897] Lustre: Skipped 2 previous similar messages [ 742.993455] Lustre: server umount lustre-MDT0000 complete [ 747.060423] LustreError: 23257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787884318 with bad export cookie 462762643850106126 [ 747.067463] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 747.078374] LustreError: 23257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 747.681657] Lustre: server umount lustre-MDT0001 complete [ 758.273782] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 22:32:07 (1787884327) [ 777.012859] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 790.211334] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 835.494648] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 847.192164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 847.524276] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 847.558654] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 847.648330] Lustre: lustre-MDT0000: new disk, initializing [ 847.752733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 847.766068] Lustre: Skipped 2 previous similar messages [ 847.789792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 853.584362] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 865.551677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 865.760522] Lustre: 29652:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 866.107439] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 871.945420] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 882.783086] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 890.027538] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 898.628233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 899.145539] Lustre: 31622:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 901.066804] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 901.137340] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 901.266078] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 901.290987] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 907.281886] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 918.674206] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 926.709586] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 952.299936] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 952.304562] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 952.310930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 952.349061] Lustre: Skipped 4 previous similar messages [ 952.368032] Lustre: Skipped 3 previous similar messages [ 955.862531] Lustre: server umount lustre-OST0000 complete [ 966.600932] Lustre: server umount lustre-MDT0000 complete [ 967.651761] LustreError: 29664:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 967.675436] LustreError: 29664:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [ 970.397750] LustreError: 29646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787884541 with bad export cookie 462762643850106532 [ 970.400101] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 970.409123] LustreError: 29646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 970.926265] Lustre: server umount lustre-MDT0001 complete [ 981.517898] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 984.924221] Key type lgssc unregistered [ 985.242797] LNet: 33607:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 985.260560] LNetError: 33607:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 985.284221] LNet: Removed LNI 192.168.203.144@tcp [ 986.802262] Key type .llcrypt unregistered [ 986.804463] Key type ._llcrypt unregistered [ 1018.030762] Key type ._llcrypt registered [ 1018.035099] Key type .llcrypt registered [ 1018.266272] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 1019.741339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1019.774306] alg: No test for adler32 (adler32-zlib) [ 1020.953882] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 1021.236705] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 1022.999684] Key type lgssc registered [ 1024.483446] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1034.668233] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1034.693602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1036.561985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1041.592743] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1045.989616] LustreError: 35023:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1046.017359] LustreError: 35023:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1050.978387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1051.107064] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1051.251504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1055.758780] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1064.366286] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1070.720389] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1077.939417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1078.422463] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1081.561384] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1085.557140] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1095.299238] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1095.860200] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1112.158184] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 22:38:01 (1787884681) [ 1114.762826] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1117.141441] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1126.864155] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 22:38:15 (1787884695) [ 1137.542920] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 22:38:26 (1787884706) [ 1138.731658] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1148.786188] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1148.789477] Lustre: Skipped 2 previous similar messages [ 1150.741346] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 22:38:40 (1787884720) [ 1152.810596] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1152.816462] Lustre: Skipped 2 previous similar messages [ 1161.881420] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1161.885859] Lustre: Skipped 4 previous similar messages [ 1177.692629] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 22:39:07 (1787884747) [ 1180.866781] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1184.683416] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1194.423825] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 22:39:23 (1787884763) [ 1196.477730] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1196.483570] Lustre: Skipped 6 previous similar messages [ 1211.963674] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 22:39:41 (1787884781) [ 1229.284184] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1229.289111] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1229.300226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1230.306277] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1230.317359] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1230.333715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1232.352570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1232.366354] Lustre: Skipped 1 previous similar message [ 1233.708857] Lustre: server umount lustre-OST0000 complete [ 1240.552146] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1240.563120] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1240.581449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1242.593394] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1244.105748] Lustre: server umount lustre-MDT0000 complete [ 1247.712179] LustreError: 35024:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1247.736984] LustreError: 35024:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1247.985850] LustreError: 35004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787884819 with bad export cookie 11079921762698000644 [ 1247.993457] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1247.997279] LustreError: 35004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1248.404844] Lustre: server umount lustre-MDT0001 complete [ 1260.205210] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 1263.040698] Key type lgssc unregistered [ 1263.299551] LNet: 42885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1263.312324] LNetError: 42885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1263.327717] LNet: Removed LNI 192.168.203.144@tcp [ 1264.220119] Key type .llcrypt unregistered [ 1264.222680] Key type ._llcrypt unregistered [ 1282.266728] Key type ._llcrypt registered [ 1282.269859] Key type .llcrypt registered [ 1282.348795] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 1301.102367] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 1303.175496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1303.202042] alg: No test for adler32 (adler32-zlib) [ 1304.261466] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 1304.675790] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 1306.464111] Key type lgssc registered [ 1307.735866] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1349.231513] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1349.320485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1350.633083] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1350.687118] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1350.844797] Lustre: lustre-MDT0000: new disk, initializing [ 1350.975833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1351.019312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1356.848542] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1368.916552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1369.064958] Lustre: 46043:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1369.205636] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1369.215318] Lustre: Skipped 1 previous similar message [ 1369.294543] Lustre: lustre-MDT0001: new disk, initializing [ 1369.429060] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1369.518342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1369.531041] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1375.208267] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1385.403679] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1391.461476] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1399.232835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1399.635691] Lustre: lustre-OST0000: new disk, initializing [ 1399.641119] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1399.649532] Lustre: 48011:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1399.771800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1406.175452] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1409.608135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1409.624386] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1409.812856] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1416.727368] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1421.447832] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1421.658728] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1426.151248] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1426.348780] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1430.501789] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1430.516124] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1430.545157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1430.549301] Lustre: Skipped 1 previous similar message [ 1434.469298] Lustre: server umount lustre-OST0000 complete [ 1440.737256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1440.739717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1440.752870] Lustre: Skipped 2 previous similar messages [ 1444.563675] Lustre: server umount lustre-MDT0000 complete [ 1448.212355] LustreError: 46034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787885019 with bad export cookie 5515272778002412913 [ 1448.230982] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1448.565813] Lustre: server umount lustre-MDT0001 complete [ 1456.816849] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 22:43:46 (1787885026) [ 1463.951040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1464.435926] LustreError: 50472:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1464.463843] LustreError: 50472:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 1464.558651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1469.546349] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1469.920928] LustreError: 50473:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1474.022079] LustreError: 50472:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1479.152838] LustreError: 50473:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1479.753841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1480.167736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1485.811779] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1494.228438] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1500.385738] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1507.249142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1507.584653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1515.138254] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1522.716344] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1530.482229] LustreError: 53002:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1531.640087] Lustre: Modifying parameter general.timeout=1 in log params [ 1532.735377] Lustre: Modifying parameter general.timeout=2 in log params [ 1535.602546] Lustre: Modifying parameter general.timeout=5 in log params [ 1535.608575] Lustre: Skipped 2 previous similar messages [ 1540.466874] Lustre: Modifying parameter general.timeout=10 in log params [ 1540.479615] Lustre: Skipped 4 previous similar messages [ 1548.991606] Lustre: Modifying parameter general.timeout=19 in log params [ 1548.995718] Lustre: Skipped 8 previous similar messages [ 1565.211246] Lustre: Modifying parameter general.timeout=38 in log params [ 1565.214397] Lustre: Skipped 18 previous similar messages [ 1597.281847] Lustre: Modifying parameter general.timeout=75 in log params [ 1597.288575] Lustre: Skipped 36 previous similar messages [ 1656.430918] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 22:47:05 (1787885225) [ 1659.179758] Lustre: 59953:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1660.763658] Lustre: 60051:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1661.647743] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 in log params [ 1661.660598] Lustre: Skipped 58 previous similar messages [ 1664.330539] Lustre: 60249:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1664.339176] Lustre: 60249:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1695.411300] Lustre: 61826:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1695.418970] Lustre: 61826:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1695.437477] LustreError: 61826:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1699.369248] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 22:47:48 (1787885268) [ 1715.682932] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1715.694295] LustreError: Skipped 1 previous similar message [ 1715.696550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1715.707318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1715.728504] Lustre: Skipped 1 previous similar message [ 1715.734201] Lustre: Skipped 1 previous similar message [ 1718.360689] Lustre: server umount lustre-MDT0000 complete [ 1720.803298] LustreError: 50472:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1720.808847] LustreError: 50472:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1722.066527] LustreError: 53051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787885293 with bad export cookie 5515272778002413795 [ 1722.070520] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1722.088195] LustreError: 53051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1722.403051] Lustre: server umount lustre-MDT0001 complete [ 1737.768240] Lustre: server umount lustre-OST0000 complete [ 1770.063274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1770.381713] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1770.664097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1775.862514] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1783.926375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1784.075148] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1784.118572] Lustre: 64361:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1788.776168] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1795.309411] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1800.250715] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1806.471341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1806.616472] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1806.865324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1806.870338] Lustre: Skipped 1 previous similar message [ 1813.339269] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1821.121986] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1822.803517] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1822.824431] Lustre: Skipped 19 previous similar messages [ 1822.886412] Lustre: 66928:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1830.881070] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1830.882521] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1830.893735] LustreError: Skipped 1 previous similar message [ 1830.899651] Lustre: Skipped 1 previous similar message [ 1830.904291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1830.916650] Lustre: Skipped 3 previous similar messages [ 1836.003928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1836.009809] Lustre: Skipped 1 previous similar message [ 1836.529698] Lustre: server umount lustre-OST0000 complete [ 1840.610174] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1840.616950] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1840.628539] Lustre: Skipped 1 previous similar message [ 1846.242481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1846.253078] Lustre: Skipped 4 previous similar messages [ 1846.478623] Lustre: server umount lustre-MDT0000 complete [ 1849.904917] LustreError: 64353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787885421 with bad export cookie 5515272778002416826 [ 1849.910883] LustreError: 64353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1849.913627] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1850.316581] Lustre: server umount lustre-MDT0001 complete [ 1859.654824] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 1862.383805] Key type lgssc unregistered [ 1862.616243] LNet: 68142:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1862.624379] LNetError: 68142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1862.639729] LNet: Removed LNI 192.168.203.144@tcp [ 1863.488773] Key type .llcrypt unregistered [ 1863.490955] Key type ._llcrypt unregistered [ 1887.483465] Key type ._llcrypt registered [ 1887.485549] Key type .llcrypt registered [ 1887.625109] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 1888.595431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1888.668727] alg: No test for adler32 (adler32-zlib) [ 1889.788450] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 1890.062092] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 1891.783254] Key type lgssc registered [ 1892.883791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1900.780578] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1900.815347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1902.372933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1906.832739] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1911.776914] LustreError: 69574:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1911.787332] LustreError: 69574:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1915.854432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1916.365221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1920.401097] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1927.708828] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1933.753756] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1940.330752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1940.856945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1947.954372] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1956.193843] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1964.724498] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1972.709196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1972.709294] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1972.710216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1972.723546] Lustre: Skipped 2 previous similar messages [ 1977.831983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1977.836116] Lustre: Skipped 3 previous similar messages [ 1978.856601] Lustre: server umount lustre-MDT0000 complete [ 1982.254231] LustreError: 69554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787885553 with bad export cookie 15988585895068702994 [ 1982.258380] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1982.261274] LustreError: 69554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1982.541326] Lustre: server umount lustre-MDT0001 complete [ 1995.987716] Lustre: server umount lustre-OST0000 complete [ 2023.810600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2024.220646] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2024.272165] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2024.650599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2029.907277] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2038.139288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2038.247319] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2038.285880] Lustre: 74447:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2038.752068] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2043.455230] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2051.765067] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2057.928375] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2064.633183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2064.862182] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2065.294300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2071.205987] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2078.220603] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2079.732862] Lustre: 77021:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2081.699394] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2086.882268] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2086.898357] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2086.926791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2092.520385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2092.526200] Lustre: Skipped 3 previous similar messages [ 2093.006480] Lustre: server umount lustre-OST0000 complete [ 2097.638150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2097.644200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2097.663379] Lustre: Skipped 2 previous similar messages [ 2097.687667] Lustre: Skipped 1 previous similar message [ 2103.146107] Lustre: server umount lustre-MDT0000 complete [ 2106.599620] LustreError: 77133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787885677 with bad export cookie 15988585895068703820 [ 2106.607136] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2106.610531] LustreError: 77133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2106.914780] Lustre: server umount lustre-MDT0001 complete [ 2118.721518] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 2121.653328] Key type lgssc unregistered [ 2121.961782] LNet: 78238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2121.973251] LNetError: 78238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2122.004640] LNet: Removed LNI 192.168.203.144@tcp [ 2123.034975] Key type .llcrypt unregistered [ 2123.038662] Key type ._llcrypt unregistered [ 2149.680364] Key type ._llcrypt registered [ 2149.684766] Key type .llcrypt registered [ 2149.818139] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2150.714858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2150.811099] alg: No test for adler32 (adler32-zlib) [ 2152.014491] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 2152.257661] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 2153.935161] Key type lgssc registered [ 2155.143257] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2162.583213] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2162.600920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2164.115995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2168.200047] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2174.435232] LustreError: 79651:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2174.463404] LustreError: 79651:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2177.059629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2177.677720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2182.371278] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2189.549896] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2195.116943] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2201.973932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2202.737139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2208.683777] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2216.895540] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2223.965489] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2224.834758] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2226.620527] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2226.628529] Lustre: Skipped 1 previous similar message [ 2229.122455] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2229.128160] Lustre: Skipped 2 previous similar messages [ 2233.874457] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2233.881633] Lustre: Skipped 5 previous similar messages [ 2242.323472] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2242.330282] Lustre: Skipped 8 previous similar messages [ 2258.697027] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2258.708449] Lustre: Skipped 17 previous similar messages [ 2290.751124] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2290.756515] Lustre: Skipped 35 previous similar messages [ 2321.458237] Lustre: 87756:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2325.385194] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 22:58:14 (1787885894) [ 2327.414616] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2329.536733] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 22:58:18 (1787885898) [ 2335.712037] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2335.717159] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2335.730360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2336.741251] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2336.744854] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2336.750973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2340.948710] Lustre: server umount lustre-OST0000 complete [ 2346.464485] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2346.479430] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2346.502649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2346.513401] Lustre: Skipped 1 previous similar message [ 2352.096539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2352.102482] Lustre: Skipped 2 previous similar messages [ 2357.218125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2357.221975] Lustre: Skipped 1 previous similar message [ 2359.264131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2359.450514] Lustre: server umount lustre-MDT0000 complete [ 2362.338976] LustreError: 87621:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2362.370691] LustreError: 87621:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2362.984901] LustreError: 82240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787885934 with bad export cookie 7905763712889787735 [ 2362.989853] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2362.993632] LustreError: 82240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2363.367876] Lustre: server umount lustre-MDT0001 complete [ 2374.092272] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 2376.683192] Key type lgssc unregistered [ 2376.897154] LNet: 89011:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2376.904842] LNetError: 89011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2376.916623] LNet: Removed LNI 192.168.203.144@tcp [ 2377.760948] Key type .llcrypt unregistered [ 2377.764801] Key type ._llcrypt unregistered [ 2381.760302] Key type ._llcrypt registered [ 2381.767224] Key type .llcrypt registered [ 2381.928634] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2385.866703] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules [ 2386.501099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2386.520615] alg: No test for adler32 (adler32-zlib) [ 2387.554755] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 2387.738457] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 2389.367167] Key type lgssc registered [ 2390.701918] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2391.952558] LustreError: 90213:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2408.414906] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2413.786061] LustreError: 90213:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2417.707843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2417.722800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2419.182298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2423.977668] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2429.415309] LustreError: 91237:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2429.449223] LustreError: 91237:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2433.882091] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 23:00:03 (1787886003) [ 2434.534824] LustreError: 91238:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2439.094754] LustreError: 91269:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2439.107857] LustreError: 91269:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2439.127175] LustreError: 91269:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 2439.539799] Lustre: server umount lustre-MDT0000 complete [ 2455.179251] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 2458.393887] Key type lgssc unregistered [ 2458.682664] LNet: 92766:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2458.691685] LNetError: 92766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2458.711354] LNet: Removed LNI 192.168.203.144@tcp [ 2459.612833] Key type .llcrypt unregistered [ 2459.618041] Key type ._llcrypt unregistered [ 2478.482237] Key type ._llcrypt registered [ 2478.484250] Key type .llcrypt registered [ 2478.574822] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2479.433348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2479.555500] alg: No test for adler32 (adler32-zlib) [ 2480.655752] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 2480.925189] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 2482.575159] Key type lgssc registered [ 2483.777883] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2491.896692] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2491.949062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2493.564636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2498.316828] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2503.154050] LustreError: 93997:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2503.168174] LustreError: 93997:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2506.956478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2507.595331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2512.428287] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2521.245601] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2528.034520] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2535.718739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2536.435277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2543.348439] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2551.662717] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2601.441782] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 23:02:50 (1787886170) [ 2603.173585] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2604.868534] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 23:02:54 (1787886174) [ 2609.634292] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2609.650210] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2609.669885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2613.222490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2613.222778] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2614.837535] Lustre: server umount lustre-MDT0000 complete [ 2618.200890] LustreError: 93977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787886189 with bad export cookie 12632599617794139136 [ 2618.206386] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2618.213031] LustreError: 93977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2618.336687] LustreError: 95006:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2618.337702] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2618.373149] Lustre: Skipped 1 previous similar message [ 2618.388770] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2618.394684] Lustre: Skipped 2 previous similar messages [ 2618.517733] Lustre: server umount lustre-MDT0001 complete [ 2628.338982] Lustre: server umount lustre-OST0000 complete [ 2639.043915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2639.403630] LustreError: 98291:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2639.545524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2644.101728] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2644.960313] LustreError: 98292:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2649.072058] LustreError: 98291:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2652.013873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2652.770848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2657.128600] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2664.614884] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2670.174840] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2683.748386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2683.978919] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2683.990308] LustreError: 98279:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2684.032488] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2684.054147] LustreError: 100376:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2684.063493] LustreError: 100376:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2684.072752] LustreError: 100376:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2684.211852] Lustre: server umount lustre-OST0000 complete [ 2684.223339] LustreError: 100376:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2692.812849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2693.020221] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2693.031771] LustreError: 98279:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2693.043260] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2693.054360] LustreError: 100838:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2693.070324] LustreError: 100838:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2693.075967] LustreError: 100838:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2693.159294] Lustre: server umount lustre-OST0000 complete [ 2693.162478] LustreError: 100838:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2702.627275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2702.892211] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2702.897130] Lustre: Found index 0 for lustre-OST0000, updating log [ 2702.904869] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2703.069268] Lustre: lustre-OST0000: new disk, initializing [ 2703.072337] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2703.078984] Lustre: 101373:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2703.365830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2709.387475] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2713.836818] Lustre: Failing over lustre-OST0000 [ 2714.095239] Lustre: server umount lustre-OST0000 complete [ 2719.200487] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2719.216157] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2719.232880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2726.883870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2726.894072] Lustre: Skipped 4 previous similar messages [ 2728.202252] Lustre: server umount lustre-MDT0000 complete [ 2731.794497] LustreError: 98271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787886303 with bad export cookie 12632599617794141887 [ 2731.801048] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2731.806789] LustreError: 98271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2732.126205] Lustre: server umount lustre-MDT0001 complete [ 2742.270504] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 23:05:11 (1787886311) [ 2760.909146] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2772.709034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2773.109342] LustreError: 103720:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2773.266278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2778.552351] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2788.618313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2788.836262] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2793.750476] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2796.880353] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2804.295530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2805.174822] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2811.273492] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2814.381253] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2814.396382] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2814.417829] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2819.565706] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [ 2819.591655] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2819.601468] Lustre: lustre-OST0000: Recovery over after 0:05, of 1 clients 1 recovered and 0 were evicted. [ 2819.607611] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2819.630778] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2820.017323] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2820.085389] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2820.165650] Lustre: lustre-OST0001: new disk, initializing [ 2820.170652] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2820.180079] Lustre: 106036:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2820.500035] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2820.511075] Lustre: Skipped 2 previous similar messages [ 2827.426732] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2829.989625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2829.999577] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2830.126220] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2835.333497] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2843.818518] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2850.795289] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2850.802935] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2850.806435] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2850.823814] Lustre: Skipped 2 previous similar messages [ 2854.009231] Lustre: server umount lustre-MDT0001 complete [ 2855.909457] LustreError: 103721:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2855.938416] LustreError: 103721:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 2863.403875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2863.551815] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2863.555715] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2863.586444] Lustre: 103710:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2865.603064] LustreError: 107622:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2865.616732] LustreError: 107622:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2865.633916] LustreError: 103712:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2865.723635] LustreError: 107629:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.144@tcp: cfg command failed: rc = -17 [ 2865.734033] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2865.734033] [ 2868.517303] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2869.746433] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2879.367477] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 23:07:28 (1787886448) [ 2895.573451] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2912.627795] Lustre: 109651:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2933.978692] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2938.006651] Lustre: 110786:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3033.057089] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3033.066160] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3033.083313] Lustre: Skipped 1 previous similar message [ 3033.087700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3033.091739] Lustre: Skipped 3 previous similar messages [ 3036.033192] Lustre: server umount lustre-MDT0000 complete [ 3038.688769] LustreError: 103715:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3038.714647] LustreError: 103715:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 8 previous similar messages [ 3039.699590] LustreError: 103700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787886610 with bad export cookie 12632599617794142818 [ 3039.703163] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3039.720668] LustreError: 103700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3040.038139] Lustre: server umount lustre-MDT0001 complete [ 3054.253457] Lustre: server umount lustre-OST0000 complete [ 3068.252825] Lustre: server umount lustre-OST0001 complete [ 3073.640675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3083.826720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3092.699285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3102.454131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3121.586496] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 3131.580718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3131.604217] Lustre: lustre-MDT0000: reset Object Index mappings [ 3131.956908] LustreError: 114744:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3132.148914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3132.155443] Lustre: Skipped 1 previous similar message [ 3136.935759] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3146.682511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3146.719896] Lustre: lustre-MDT0001: reset Object Index mappings [ 3153.375304] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3156.467448] Lustre: 115892:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3163.036480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3171.226949] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3172.416359] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3178.555403] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 3180.928761] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3186.695269] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3186.708090] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3189.016479] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3197.283299] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3201.161542] Lustre: 117767:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3250.597227] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 23:13:40 (1787886820) [ 3278.826519] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3278.830956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3278.849139] Lustre: Skipped 6 previous similar messages [ 3278.860216] Lustre: Skipped 6 previous similar messages [ 3282.408295] Lustre: server umount lustre-MDT0000 complete [ 3283.940256] LustreError: 116886:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3283.951599] LustreError: 116886:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [ 3286.311177] LustreError: 114725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787886857 with bad export cookie 12632599617794189907 [ 3286.314361] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3286.331188] LustreError: 114725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3292.916498] Lustre: server umount lustre-MDT0001 complete [ 3302.875393] Lustre: server umount lustre-OST0000 complete [ 3312.481093] Lustre: server umount lustre-OST0001 complete [ 3320.221430] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 3331.043500] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 3369.751747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3369.959184] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3369.994986] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3370.091457] Lustre: lustre-MDT0000: new disk, initializing [ 3370.169282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3370.176649] Lustre: Skipped 3 previous similar messages [ 3370.189646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3375.407292] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3382.531579] Lustre: server umount lustre-MDT0000 complete [ 3392.324557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3392.640770] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3392.652210] Lustre: 122039:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3392.670612] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3396.678789] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3402.677656] Lustre: server umount lustre-MDT0000 complete [ 3413.072673] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 23:16:22 (1787886982) [ 3415.656506] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3418.060080] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 23:16:27 (1787886987) [ 3435.862497] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 3446.825232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3451.720344] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3459.992534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3460.167289] Lustre: 124089:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3460.243744] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3460.247612] Lustre: Skipped 1 previous similar message [ 3460.351276] Lustre: lustre-MDT0001: new disk, initializing [ 3460.538458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3460.549033] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3464.784139] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3469.875329] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3476.563803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3476.838073] Lustre: lustre-OST0000: new disk, initializing [ 3476.843974] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3476.858618] Lustre: 125666:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3478.812576] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3478.833142] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3478.988968] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3484.410984] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3496.279832] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3496.558528] Lustre: lustre-OST0001: new disk, initializing [ 3496.562067] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3496.574345] Lustre: 126537:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3498.161578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3498.181345] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3498.303062] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3503.776744] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3513.995371] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3517.733360] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3918.930844] Lustre: 132950:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3918.936859] Lustre: 132950:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3918.941595] Lustre: 132950:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3918.947606] Lustre: 132950:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3918.954452] Lustre: 132950:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3918.959313] Lustre: 132950:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4217.683043] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 23:29:46 (1787887786) [ 4233.697100] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4233.701993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4233.714395] Lustre: Skipped 2 previous similar messages [ 4233.720948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4233.724262] Lustre: Skipped 4 previous similar messages [ 4238.840834] Lustre: server umount lustre-MDT0000 complete [ 4240.869958] LustreError: 125196:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4240.883994] LustreError: 125196:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 4242.458477] LustreError: 132962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787887813 with bad export cookie 12632599617794258290 [ 4242.468799] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4242.480234] LustreError: 132962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4245.998234] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4246.014680] Lustre: Skipped 4 previous similar messages [ 4249.010575] Lustre: server umount lustre-MDT0001 complete [ 4259.094324] Lustre: server umount lustre-OST0000 complete [ 4277.219406] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4277.485662] Lustre: server umount lustre-OST0001 complete [ 4285.616628] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 4296.343057] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 4336.860900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4337.259606] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4337.293989] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4337.405385] Lustre: lustre-MDT0000: new disk, initializing [ 4337.518429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4337.529786] Lustre: Skipped 5 previous similar messages [ 4337.558139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4341.631564] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4351.909344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4352.013950] Lustre: 136018:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4352.128312] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4352.140124] Lustre: Skipped 1 previous similar message [ 4352.265876] Lustre: lustre-MDT0001: new disk, initializing [ 4352.410985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4352.422962] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4356.568605] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4365.623483] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4370.774734] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4377.446131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4377.747770] Lustre: lustre-OST0000: new disk, initializing [ 4377.752339] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4377.760987] Lustre: 137984:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4377.838580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4377.843121] Lustre: Skipped 1 previous similar message [ 4379.356428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4379.367252] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4379.478514] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4383.454257] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4391.936170] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4400.836926] Lustre: lustre-MDD0000: changelog on [ 4403.770104] Lustre: lustre-MDD0001: changelog on [ 6438.163629] Lustre: lustre-MDD0001: changelog off [ 6440.026536] Lustre: 138642:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6440.859449] Lustre: lustre-MDD0000: changelog off [ 6448.100363] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6448.101267] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6448.104954] LustreError: Skipped 1 previous similar message [ 6448.115297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6448.118781] Lustre: Skipped 9 previous similar messages [ 6452.571692] Lustre: server umount lustre-OST0000 complete [ 6456.801108] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6456.810381] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6456.827146] Lustre: Skipped 1 previous similar message [ 6456.844083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6462.920425] Lustre: server umount lustre-MDT0000 complete [ 6463.458828] LustreError: 136029:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6463.474065] LustreError: 136029:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 6467.373107] LustreError: 136010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787890038 with bad export cookie 12632599617795076002 [ 6467.375946] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6467.393791] LustreError: 136010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6467.669806] Lustre: server umount lustre-MDT0001 complete [ 6478.966673] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 6482.179083] Key type lgssc unregistered [ 6482.582227] LNet: 147165:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6482.587938] LNetError: 147165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6482.606073] LNet: Removed LNI 192.168.203.144@tcp [ 6483.685366] Key type .llcrypt unregistered [ 6483.689615] Key type ._llcrypt unregistered [ 6503.545795] Key type ._llcrypt registered [ 6503.549524] Key type .llcrypt registered [ 6503.682943] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 6517.601587] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 6518.751244] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6518.797578] alg: No test for adler32 (adler32-zlib) [ 6519.871670] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 6520.116811] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 6521.791207] Key type lgssc registered [ 6522.867870] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6554.941425] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6554.987571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6556.293879] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6556.331651] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6556.410702] Lustre: lustre-MDT0000: new disk, initializing [ 6556.481952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6556.501434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6560.392102] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6570.198038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6570.313145] Lustre: 150364:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6570.381095] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6570.390047] Lustre: Skipped 1 previous similar message [ 6570.482721] Lustre: lustre-MDT0001: new disk, initializing [ 6570.581831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6570.635081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6570.645315] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6574.450306] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6582.698377] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6587.546369] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6593.474801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6593.809871] Lustre: lustre-OST0000: new disk, initializing [ 6593.814182] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6593.819870] Lustre: 152334:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6593.912976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6600.319329] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6601.795049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6601.815233] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6601.959658] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6609.613967] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6615.875743] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_module obdecho/obdecho [ 6617.616690] LustreError: 153336:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6617.627715] LustreError: 153336:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6619.237950] LustreError: 153432:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6631.905942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6631.910524] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6631.921103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6632.933587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6632.934060] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6634.760691] Lustre: server umount lustre-MDT0000 complete [ 6638.347768] LustreError: 150358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787890209 with bad export cookie 7580222170570704164 [ 6638.350716] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6638.358308] LustreError: 150358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6638.720782] Lustre: server umount lustre-MDT0001 complete [ 6652.524980] Lustre: server umount lustre-OST0000 complete [ 6673.628899] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 6682.904745] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 6712.382726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6712.649425] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6712.708140] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6712.797250] Lustre: lustre-MDT0000: new disk, initializing [ 6712.873341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6712.902218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6717.256710] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6727.091079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6727.220923] Lustre: 157152:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6727.289750] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6727.295994] Lustre: Skipped 1 previous similar message [ 6727.378183] Lustre: lustre-MDT0001: new disk, initializing [ 6727.486570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6727.535735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6727.556536] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6731.156937] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6739.020794] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6743.668691] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6748.923348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6749.098751] Lustre: lustre-OST0000: new disk, initializing [ 6749.102373] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6749.107708] Lustre: 159119:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6749.179458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6754.924072] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6756.420169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6756.426266] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6756.533546] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6764.047152] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6767.600928] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6767.757790] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6771.061714] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6771.271604] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6777.314278] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6777.316785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6777.317937] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6777.332170] Lustre: Skipped 2 previous similar messages [ 6777.339885] Lustre: Skipped 2 previous similar messages [ 6782.437330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6782.446446] Lustre: Skipped 2 previous similar messages [ 6787.039188] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6787.211810] Lustre: server umount lustre-OST0000 complete [ 6792.675205] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6792.696707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6792.704346] Lustre: Skipped 1 previous similar message [ 6796.329464] Lustre: server umount lustre-MDT0000 complete [ 6797.797079] LustreError: 158063:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6797.815226] LustreError: 158063:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 6799.495146] LustreError: 157143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787890370 with bad export cookie 7580222170570705123 [ 6799.500893] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6799.513269] LustreError: 157143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6799.763195] Lustre: server umount lustre-MDT0001 complete [ 6802.552737] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 00:12:52 (1787890372) [ 6803.960555] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6805.477433] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 00:12:55 (1787890375) [ 6812.204466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6812.636550] LustreError: 161532:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6812.695043] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6816.607287] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6817.761972] LustreError: 161533:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6821.860053] LustreError: 161532:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6824.091776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6828.427269] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6834.901421] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6839.645674] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6845.445704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6845.917656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6845.925606] Lustre: Skipped 1 previous similar message [ 6851.104418] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6858.611670] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6864.519494] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.44@tcp (stopping) [ 6865.379668] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6865.395266] LustreError: Skipped 1 previous similar message [ 6865.398426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6865.413240] Lustre: Skipped 1 previous similar message [ 6870.575567] Lustre: server umount lustre-MDT0000 complete [ 6873.575507] LustreError: 163551:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6873.594215] LustreError: 163551:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6873.978327] LustreError: 161514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787890445 with bad export cookie 7580222170570706005 [ 6873.980668] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6873.998310] LustreError: 161514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6878.696428] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6878.714018] Lustre: Skipped 2 previous similar messages [ 6880.445571] Lustre: server umount lustre-MDT0001 complete [ 6887.113196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6899.196036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6899.433714] LustreError: 165372:0:(llog.c:1655:llog_backup()) MGC192.168.203.144@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6899.438821] Lustre: 165372:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.144@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6905.317285] LustreError: 165377:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6905.333963] LustreError: 165377:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 6921.819553] LustreError: 165379:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6921.850035] LustreError: 165379:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 13 previous similar messages [ 6956.002720] LustreError: 165398:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6956.029994] LustreError: 165398:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 23 previous similar messages [ 6960.610837] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6960.619971] Lustre: 165372:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.144@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7001.610152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7006.424953] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7006.699418] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7015.284761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7020.009228] LustreError: 165378:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7020.038880] LustreError: 165378:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 45 previous similar messages [ 7025.183903] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7025.657481] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7032.471826] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7034.344743] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7040.357545] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7041.815973] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7061.475810] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7061.477631] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7061.480557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7061.480568] Lustre: Skipped 8 previous similar messages [ 7061.487345] Lustre: Skipped 1 previous similar message [ 7066.225127] Lustre: server umount lustre-MDT0000 complete [ 7070.203344] LustreError: 161513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787890641 with bad export cookie 7580222170570712459 [ 7070.210226] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7070.218092] LustreError: 161513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7070.630748] Lustre: server umount lustre-MDT0001 complete [ 7085.357361] Lustre: server umount lustre-OST0000 complete [ 7105.926132] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 7114.911312] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 7151.647420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7152.103857] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7152.135927] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7152.243847] Lustre: lustre-MDT0000: new disk, initializing [ 7152.384808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7152.406615] Lustre: Skipped 1 previous similar message [ 7152.466556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7157.429163] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7168.335695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7168.479536] Lustre: 170477:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7168.775640] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7172.966352] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7182.701388] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7189.558641] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7197.218794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7197.688274] Lustre: lustre-OST0000: new disk, initializing [ 7197.691966] Lustre: Skipped 1 previous similar message [ 7197.697807] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7197.705103] Lustre: Skipped 2 previous similar messages [ 7197.709522] Lustre: 172448:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7204.090297] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7207.548316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7207.567626] Lustre: Skipped 1 previous similar message [ 7207.584190] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7207.745134] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7213.838614] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7218.721332] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7218.955820] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7223.409923] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7223.708471] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7228.385732] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7228.408872] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7228.436872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7228.451072] Lustre: Skipped 3 previous similar messages [ 7231.974531] Lustre: server umount lustre-OST0000 complete [ 7242.056972] Lustre: server umount lustre-MDT0000 complete [ 7243.745675] LustreError: 171642:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7243.770293] LustreError: 171642:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 7245.739114] LustreError: 170471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787890817 with bad export cookie 7580222170570729294 [ 7245.741819] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7245.758025] LustreError: 170471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7246.200643] Lustre: server umount lustre-MDT0001 complete [ 7254.401743] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 00:20:24 (1787890824) [ 7261.221107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7267.078669] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7276.037878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7280.677394] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7287.557858] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7293.557895] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7300.880675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7307.390462] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7314.946262] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7324.987090] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 00:21:34 (1787890894) [ 7332.320870] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7332.327994] LustreError: Skipped 1 previous similar message [ 7332.332267] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7332.345590] Lustre: Skipped 3 previous similar messages [ 7332.353536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7332.358567] Lustre: Skipped 2 previous similar messages [ 7334.789979] Lustre: server umount lustre-OST0000 complete [ 7344.342189] Lustre: server umount lustre-MDT0000 complete [ 7347.764780] LustreError: 174839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787890919 with bad export cookie 7580222170570730176 [ 7347.768836] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7347.773131] LustreError: 174839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7348.106072] Lustre: server umount lustre-MDT0001 complete [ 7359.272186] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 7361.996595] Key type lgssc unregistered [ 7362.315315] LNet: 178592:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7362.332301] LNetError: 178592:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7362.356465] LNet: Removed LNI 192.168.203.144@tcp [ 7363.286891] Key type .llcrypt unregistered [ 7363.290404] Key type ._llcrypt unregistered [ 7379.748873] Key type ._llcrypt registered [ 7379.751725] Key type .llcrypt registered [ 7379.862222] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 7380.627763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7380.657851] alg: No test for adler32 (adler32-zlib) [ 7381.674949] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 7381.912145] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 7383.575160] Key type lgssc registered [ 7384.569139] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7391.282428] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7391.305055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7406.815485] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7411.935376] LustreError: 179829:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.144@tcp: failed processing log, type 4: rc = -110 [ 7417.055726] LustreError: 179829:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7417.070508] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7417.075739] LustreError: MGC192.168.203.144@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7417.083262] LustreError: 179829:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7417.091474] LustreError: 179829:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7417.095642] LustreError: 179829:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7417.142777] Lustre: server umount lustre-OST0000 complete [ 7417.145664] LustreError: 179829:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7425.212526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7426.551888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7431.328650] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7435.748087] LustreError: 180328:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7435.772333] LustreError: 180328:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7440.744854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7440.865780] LustreError: 180329:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7441.134708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7446.486277] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7453.658183] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7459.711534] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7466.778422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7466.996931] LustreError: 182144:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7467.022722] Lustre: 182144:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.144@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7467.164556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7472.492427] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7478.911608] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7481.826783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7481.831409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7481.845657] Lustre: Skipped 1 previous similar message [ 7481.862702] Lustre: Skipped 2 previous similar messages [ 7485.988072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7486.001431] Lustre: Skipped 2 previous similar messages [ 7491.043763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7491.050740] Lustre: Skipped 2 previous similar messages [ 7495.135224] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7495.399195] Lustre: server umount lustre-MDT0000 complete [ 7496.174800] LustreError: 180328:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7496.205933] LustreError: 180328:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7499.275724] LustreError: 182172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787891070 with bad export cookie 12264926647302802424 [ 7499.280564] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7499.288418] LustreError: 182172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7500.383319] LustreError: 181058:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000008dd94bfa x1874739543962112/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7500.686676] Lustre: server umount lustre-MDT0001 complete [ 7515.568830] Lustre: server umount lustre-OST0000 complete [ 7524.535146] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 7534.149253] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 7568.752270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7568.912033] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7568.933532] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7569.003597] Lustre: lustre-MDT0000: new disk, initializing [ 7569.096208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7569.116672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7573.650661] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7587.063915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7587.261988] Lustre: 185669:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7587.318928] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7587.323082] Lustre: Skipped 1 previous similar message [ 7587.425819] Lustre: lustre-MDT0001: new disk, initializing [ 7587.582139] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7587.724554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7587.750157] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7593.420653] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7602.941785] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7608.479626] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7615.075033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7615.329949] Lustre: lustre-OST0000: new disk, initializing [ 7615.339295] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7615.347479] Lustre: 187638:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7615.446720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7621.206023] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7621.220994] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7621.343028] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7622.116972] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7633.609463] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7639.076747] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7639.303362] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7643.611675] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7643.834755] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7647.208193] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7647.214566] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7647.226335] LustreError: Skipped 1 previous similar message [ 7647.241666] Lustre: Skipped 1 previous similar message [ 7647.244276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7647.263992] Lustre: Skipped 1 previous similar message [ 7652.324352] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7652.337493] Lustre: Skipped 1 previous similar message [ 7659.999300] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7660.214214] Lustre: server umount lustre-OST0000 complete [ 7664.611524] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7664.620920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7664.645968] Lustre: Skipped 1 previous similar message [ 7664.653088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7664.656163] Lustre: Skipped 2 previous similar messages [ 7667.695606] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7670.178330] Lustre: server umount lustre-MDT0000 complete [ 7672.801646] LustreError: 187087:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7672.818329] LustreError: 187087:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7673.456290] LustreError: 185661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787891244 with bad export cookie 12264926647302803145 [ 7673.464023] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7673.472839] LustreError: 185661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7673.745625] Lustre: server umount lustre-MDT0001 complete [ 7682.598606] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 00:27:32 (1787891252) [ 7698.579568] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 7701.335893] Key type lgssc unregistered [ 7701.626776] LNet: 190589:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7701.640229] LNetError: 190589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7701.649788] LNet: Removed LNI 192.168.203.144@tcp [ 7702.520770] Key type .llcrypt unregistered [ 7702.528285] Key type ._llcrypt unregistered [ 7720.968773] Key type ._llcrypt registered [ 7720.971913] Key type .llcrypt registered [ 7721.091873] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 7722.190524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7722.232027] alg: No test for adler32 (adler32-zlib) [ 7723.353686] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 7723.663361] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 7725.391457] Key type lgssc registered [ 7726.491473] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7733.448901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7733.463239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7734.978212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7739.542408] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7745.509768] LustreError: 191810:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7745.535480] LustreError: 191810:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7747.497092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7747.926815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7752.632587] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7760.939660] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7766.156629] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7774.142102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7774.445746] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7774.453209] LustreError: 193674:0:(llog.c:1696:llog_backup()) MGC192.168.203.144@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7774.466287] Lustre: 193674:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.144@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7774.593249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7781.664747] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7788.982468] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7793.386919] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7793.398829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7793.416675] Lustre: Skipped 1 previous similar message [ 7793.425959] Lustre: Skipped 1 previous similar message [ 7797.744580] Lustre: server umount lustre-MDT0000 complete [ 7798.754608] LustreError: 191809:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7798.766188] LustreError: 191809:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7801.341318] LustreError: 191789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787891372 with bad export cookie 10090372049609447150 [ 7801.345894] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7801.357070] LustreError: 191789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7801.655871] LustreError: 192540:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000000c6dd595 x1874739901526656/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7801.832791] Lustre: server umount lustre-MDT0001 complete [ 7816.069416] Lustre: server umount lustre-OST0000 complete [ 7827.029501] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 7838.119971] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 7875.747823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7876.074941] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7876.110327] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7876.235864] Lustre: lustre-MDT0000: new disk, initializing [ 7876.330163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7876.358743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7880.943757] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7891.106913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7891.214772] Lustre: 197199:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7891.282591] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7891.284657] Lustre: Skipped 1 previous similar message [ 7891.291905] LustreError: 197981:0:(llog.c:1696:llog_backup()) MGC192.168.203.144@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7891.303949] LustreError: 197981:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7891.311915] Lustre: 197981:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.144@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7891.340390] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7891.345276] Lustre: Skipped 1 previous similar message [ 7891.447533] Lustre: lustre-MDT0001: new disk, initializing [ 7891.578050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7891.661034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7891.673694] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7895.913525] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7905.225607] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7910.756633] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7917.114684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7917.298691] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7917.304507] Lustre: Skipped 1 previous similar message [ 7917.307445] LustreError: 199143:0:(llog.c:1696:llog_backup()) MGC192.168.203.144@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7917.311824] LustreError: 199143:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7917.318717] Lustre: 199143:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.144@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7917.324114] Lustre: 199143:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7917.340311] Lustre: lustre-OST0000: new disk, initializing [ 7917.343317] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7917.348654] Lustre: 199166:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7917.426028] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7923.266411] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7926.347385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7926.356744] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7926.457119] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7932.990980] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7936.776466] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7937.017310] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7941.705938] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7941.909629] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7947.242678] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7947.245606] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7947.249294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7947.249301] Lustre: Skipped 1 previous similar message [ 7947.283911] Lustre: Skipped 2 previous similar messages [ 7950.418507] Lustre: server umount lustre-OST0000 complete [ 7957.479820] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7957.489432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7957.493373] Lustre: Skipped 1 previous similar message [ 7957.517531] Lustre: Skipped 2 previous similar messages [ 7960.601847] Lustre: server umount lustre-MDT0000 complete [ 7962.608425] LustreError: 198093:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7962.649406] LustreError: 198093:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7965.402750] LustreError: 197191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787891536 with bad export cookie 10090372049609447871 [ 7965.415083] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7965.423665] LustreError: 197191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7965.925176] Lustre: server umount lustre-MDT0001 complete [ 7976.373475] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 00:32:25 (1787891545) [ 7992.713096] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 8002.692714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8002.989210] LustreError: 202294:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8003.027811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8006.825901] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8008.161128] LustreError: 202295:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8008.180868] LustreError: 202295:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8015.598258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8020.525292] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8023.545684] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8030.681845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8037.693865] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8045.949754] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8046.059896] Lustre: lustre-OST0001: new disk, initializing [ 8046.064980] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8046.087297] Lustre: 204602:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8046.201270] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8046.204465] Lustre: Skipped 2 previous similar messages [ 8052.880701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8052.907905] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8053.055733] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8053.125085] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8064.068907] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8070.313430] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8072.291419] Lustre: DEBUG MARKER: ADD OST3 [ 8077.016949] loop: module loaded [ 8077.106923] loop0: detected capacity change from 0 to 204800000 [ 8077.383912] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8077.398941] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8077.412722] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8082.717917] loop0: detected capacity change from 0 to 204800000 [ 8083.814984] Lustre: DEBUG MARKER: STOP OST3 [ 8087.993541] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8088.246665] Lustre: lustre-OST0003: new disk, initializing [ 8088.257226] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8088.273597] Lustre: 206581:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8090.294612] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8090.395614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8090.401780] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8090.435745] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8092.082433] Lustre: Failing over lustre-OST0003 [ 8092.211106] Lustre: server umount lustre-OST0003 complete [ 8092.325973] LustreError: 206819:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8094.323246] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8095.357627] LustreError: 203780:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8095.379054] LustreError: 203780:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8095.721172] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8095.724267] LustreError: Skipped 1 previous similar message [ 8095.731471] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8103.730143] Lustre: DEBUG MARKER: START OST3 again [ 8110.470194] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8110.752087] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8110.758726] Lustre: Skipped 1 previous similar message [ 8110.771739] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8112.509527] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8112.646191] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8112.647513] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8116.837862] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8135.397100] Lustre: Failing over lustre-OST0003 [ 8135.495580] Lustre: server umount lustre-OST0003 complete [ 8136.685569] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8136.695122] LustreError: 203783:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8136.702862] Lustre: Skipped 2 previous similar messages [ 8136.714481] LustreError: 203783:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 8 previous similar messages [ 8154.090650] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8154.100833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8154.120520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8158.495575] Lustre: server umount lustre-MDT0000 complete [ 8161.710640] LustreError: 202276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787891732 with bad export cookie 10090372049609448753 [ 8161.720680] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8161.744932] LustreError: 202276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8162.281227] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8162.289456] Lustre: Skipped 5 previous similar messages [ 8171.489041] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8171.493170] LustreError: 202318:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8171.500796] Lustre: Skipped 3 previous similar messages [ 8171.524203] LustreError: 202318:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 13 previous similar messages [ 8176.095943] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8176.400092] Lustre: server umount lustre-MDT0001 complete [ 8179.924780] Lustre: server umount lustre-OST0000 complete [ 8183.498550] Lustre: server umount lustre-OST0001 complete [ 8189.997418] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 8197.472849] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 8231.842056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8232.022198] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8232.048797] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8232.145715] Lustre: lustre-MDT0000: new disk, initializing [ 8232.220803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8235.900422] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8245.423036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8245.512891] Lustre: 212059:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8245.714185] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8245.719609] Lustre: Skipped 1 previous similar message [ 8245.753507] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8249.993716] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8259.093263] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8265.483499] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8272.436796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8272.692594] Lustre: lustre-OST0000: new disk, initializing [ 8272.697033] Lustre: Skipped 1 previous similar message [ 8272.701253] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8272.709569] Lustre: Skipped 2 previous similar messages [ 8272.715249] Lustre: 214028:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8274.048941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8274.062281] Lustre: Skipped 1 previous similar message [ 8274.070681] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8274.194660] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8278.752382] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8288.428692] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8293.916175] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8294.216680] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8299.583447] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8299.823967] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8305.127322] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8305.131170] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8305.149126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8305.172541] Lustre: Skipped 6 previous similar messages [ 8305.198509] Lustre: Skipped 3 previous similar messages [ 8316.383235] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8316.670626] Lustre: server umount lustre-OST0000 complete [ 8322.532726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8322.549701] LustreError: Skipped 1 previous similar message [ 8327.431721] Lustre: server umount lustre-MDT0000 complete [ 8330.728557] LustreError: 212068:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8330.762735] LustreError: 212068:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 8331.487279] LustreError: 212051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787891902 with bad export cookie 10090372049609451042 [ 8331.494639] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8331.497631] LustreError: 212051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8331.915996] Lustre: server umount lustre-MDT0001 complete [ 8335.997785] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 00:38:25 (1787891905) [ 8352.175597] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 8361.198871] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 8393.388590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8393.616543] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8393.641658] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8393.730681] Lustre: lustre-MDT0000: new disk, initializing [ 8393.871565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8398.051848] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8407.549609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8407.689466] Lustre: 218639:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8408.056982] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8412.281773] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8421.103978] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8426.121994] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8432.286093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8432.568333] Lustre: 220607:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8433.805574] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8438.221879] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8447.731266] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8452.863977] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8453.051517] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8458.069579] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8458.294718] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8464.352500] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8464.366826] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8464.379178] Lustre: Skipped 2 previous similar messages [ 8464.391206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8464.393981] Lustre: Skipped 7 previous similar messages [ 8466.432987] Lustre: server umount lustre-OST0000 complete [ 8476.451367] Lustre: server umount lustre-MDT0000 complete [ 8480.229634] LustreError: 218650:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8480.255980] LustreError: 218650:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8481.414553] LustreError: 219433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787892052 with bad export cookie 10090372049609451931 [ 8481.420325] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8481.441133] LustreError: 219433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8481.877928] Lustre: server umount lustre-MDT0001 complete [ 8493.005345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8499.223708] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8507.167044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8507.511576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8507.517644] Lustre: Skipped 5 previous similar messages [ 8511.862384] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8520.223050] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8526.226698] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8532.781344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8540.281154] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8549.312186] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8563.175768] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8563.176212] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8563.187720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8563.187729] Lustre: Skipped 5 previous similar messages [ 8563.211019] Lustre: Skipped 4 previous similar messages [ 8563.256819] LustreError: Skipped 2 previous similar messages [ 8565.957241] Lustre: server umount lustre-OST0000 complete [ 8575.627894] Lustre: server umount lustre-MDT0000 complete [ 8579.395268] LustreError: 222802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787892150 with bad export cookie 10090372049609452806 [ 8579.402166] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8579.408149] LustreError: 222802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8579.833342] Lustre: server umount lustre-MDT0001 complete [ 8689.531740] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 8693.479789] Key type lgssc unregistered [ 8693.925740] LNet: 226488:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8693.938605] LNetError: 226488:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8693.978900] LNet: Removed LNI 192.168.203.144@tcp [ 8695.226320] Key type .llcrypt unregistered [ 8695.232257] Key type ._llcrypt unregistered [ 8722.919565] Key type ._llcrypt registered [ 8722.922359] Key type .llcrypt registered [ 8723.048487] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 8742.517142] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 8743.527173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8743.731905] alg: No test for adler32 (adler32-zlib) [ 8744.883814] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 8745.137331] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 8746.815111] Key type lgssc registered [ 8747.862474] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8782.269864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8782.292311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8783.581839] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8783.637868] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8783.696935] Lustre: lustre-MDT0000: new disk, initializing [ 8783.816978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8783.853299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8788.547419] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8798.222469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8798.357894] Lustre: 229842:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8798.443662] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8798.451302] Lustre: Skipped 1 previous similar message [ 8798.529402] Lustre: lustre-MDT0001: new disk, initializing [ 8798.633834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8798.690938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8798.711463] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8803.689204] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8813.652636] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8819.950326] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8826.817721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8827.085471] Lustre: lustre-OST0000: new disk, initializing [ 8827.090194] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8827.101880] Lustre: 231812:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8827.179202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8833.420769] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8836.199095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8836.212851] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8836.413701] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8842.711625] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8847.123772] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8847.300962] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8851.677321] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8851.882815] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8857.057776] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8857.061292] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8857.074778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8857.079215] Lustre: Skipped 1 previous similar message [ 8859.784901] Lustre: server umount lustre-OST0000 complete [ 8865.252478] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8865.266846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8865.295289] Lustre: Skipped 1 previous similar message [ 8865.307554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8867.300968] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8867.306991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8867.325554] Lustre: Skipped 1 previous similar message [ 8869.975838] Lustre: server umount lustre-MDT0000 complete [ 8873.639378] LustreError: 230646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787892444 with bad export cookie 5675094413034140874 [ 8873.642721] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8873.652177] LustreError: 230646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8874.112494] Lustre: server umount lustre-MDT0001 complete [ 8891.065385] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 8900.873339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8901.301332] LustreError: 234744:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8901.337693] LustreError: 234744:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 8901.354032] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8906.181537] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8906.728680] LustreError: 234745:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8910.818880] LustreError: 234744:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8915.112671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8915.571466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8920.398565] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8923.564501] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8930.470978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8930.881648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8936.366680] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8944.797162] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8944.964830] Lustre: lustre-OST0001: new disk, initializing [ 8944.975634] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8944.981091] Lustre: 237051:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8953.043872] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8953.100802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8953.112493] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8953.261140] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8964.479572] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8973.035263] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8987.633433] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 00:49:17 (1787892557) [ 8992.229768] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8992.251348] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8992.271775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8996.617136] Lustre: server umount lustre-MDT0000 complete [ 8999.398490] LustreError: 237628:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8999.418074] LustreError: 237628:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 9000.387439] LustreError: 237050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787892571 with bad export cookie 5675094413034141756 [ 9000.404654] LustreError: 237050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9000.406527] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9000.755589] Lustre: server umount lustre-MDT0001 complete [ 9013.811668] Lustre: server umount lustre-OST0000 complete [ 9027.696565] Lustre: server umount lustre-OST0001 complete [ 9033.380259] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 9043.526029] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 9078.043208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9078.231404] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9078.266456] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9078.345961] Lustre: lustre-MDT0000: new disk, initializing [ 9078.400936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9078.405707] Lustre: Skipped 1 previous similar message [ 9078.419798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9082.833890] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9094.018903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9094.137927] Lustre: 240996:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9094.200638] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9094.205194] Lustre: Skipped 1 previous similar message [ 9094.258422] Lustre: lustre-MDT0001: new disk, initializing [ 9094.391424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9094.400944] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9098.483533] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9107.891706] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9113.958723] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9120.597037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9120.828739] Lustre: lustre-OST0000: new disk, initializing [ 9120.831946] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9120.837781] Lustre: 242965:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9120.917280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9120.922758] Lustre: Skipped 1 previous similar message [ 9122.871130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9122.892384] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9122.983489] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9126.887455] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9136.900576] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9141.675725] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9141.852745] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9146.139289] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9146.316911] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9148.903141] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9148.905363] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9148.910381] LustreError: Skipped 1 previous similar message [ 9148.922176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9148.923259] Lustre: Skipped 4 previous similar messages [ 9148.953787] Lustre: Skipped 5 previous similar messages [ 9159.152022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9159.165863] Lustre: Skipped 3 previous similar messages [ 9162.207074] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9162.487866] Lustre: server umount lustre-OST0000 complete [ 9166.306924] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9166.318323] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9172.126424] Lustre: server umount lustre-MDT0000 complete [ 9174.497547] LustreError: 241874:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9174.512066] LustreError: 241874:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9175.285311] LustreError: 240990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787892746 with bad export cookie 5675094413034143891 [ 9175.286542] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9175.293417] LustreError: 240990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9175.639284] Lustre: server umount lustre-MDT0001 complete [ 9183.371414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9183.783709] LustreError: 245182:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9188.490388] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9195.784642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9196.178679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9196.187481] Lustre: Skipped 1 previous similar message [ 9200.007124] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9207.194910] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9212.840492] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9219.725552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9227.770660] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9236.415284] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9384.928276] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9384.934991] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9384.945764] Lustre: Skipped 1 previous similar message [ 9384.953884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9384.964570] Lustre: Skipped 3 previous similar messages [ 9390.051758] Lustre: server umount lustre-OST0000 complete [ 9400.320639] Lustre: server umount lustre-MDT0000 complete [ 9402.336636] LustreError: 245910:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9402.363109] LustreError: 245910:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 9404.288238] LustreError: 245162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787892975 with bad export cookie 5675094413034144773 [ 9404.295129] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9404.313550] LustreError: 245162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9404.862158] Lustre: server umount lustre-MDT0001 complete [ 9504.084891] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 9507.130867] Key type lgssc unregistered [ 9507.506798] LNet: 248796:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9507.518334] LNetError: 248796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9507.535605] LNet: Removed LNI 192.168.203.144@tcp [ 9508.393332] Key type .llcrypt unregistered [ 9508.405189] Key type ._llcrypt unregistered [ 9530.571675] Key type ._llcrypt registered [ 9530.573817] Key type .llcrypt registered [ 9530.659468] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 9545.699043] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 9546.618951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9546.769579] alg: No test for adler32 (adler32-zlib) [ 9547.762716] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 9548.009596] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 9549.647155] Key type lgssc registered [ 9550.454981] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9594.873806] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 9605.121615] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9605.156246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9606.496957] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9606.576483] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9606.710833] Lustre: lustre-MDT0000: new disk, initializing [ 9606.802367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9606.824482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9611.305729] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9621.760700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9621.872244] Lustre: 252866:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9621.922822] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9621.927561] Lustre: Skipped 1 previous similar message [ 9621.991635] Lustre: lustre-MDT0001: new disk, initializing [ 9622.105035] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9622.144661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9622.162241] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9626.837108] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9631.449920] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9638.237808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9638.656275] Lustre: lustre-OST0000: new disk, initializing [ 9638.665120] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9638.677945] Lustre: 254495:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9638.849385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9644.106618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9644.117712] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9644.220633] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9644.427590] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9654.795862] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9655.022118] Lustre: lustre-OST0001: new disk, initializing [ 9655.031605] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9655.046785] Lustre: 255368:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9655.163056] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9661.134975] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9663.034786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9663.063885] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9663.147649] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9671.737624] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9678.313257] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9693.669402] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9693.678443] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9693.705656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9694.184710] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9694.207047] Lustre: Skipped 1 previous similar message [ 9694.221924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9694.228519] Lustre: Skipped 3 previous similar messages [ 9697.847987] Lustre: server umount lustre-MDT0000 complete [ 9702.205920] LustreError: 256192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787893273 with bad export cookie 12118997243259077799 [ 9702.210242] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9702.223190] LustreError: 256192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9702.743272] Lustre: server umount lustre-MDT0001 complete [ 9715.951165] Lustre: server umount lustre-OST0000 complete [ 9729.790937] Lustre: server umount lustre-OST0001 complete [ 9743.550407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9752.755957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9760.280478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9770.115179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9788.660380] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 9798.969729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9798.997630] Lustre: lustre-MDT0000: reset Object Index mappings [ 9799.488590] LustreError: 260390:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9799.504886] LustreError: 260390:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 9799.578632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9803.955438] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9804.773146] LustreError: 260391:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9808.876565] LustreError: 260390:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9812.980175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9813.003020] Lustre: lustre-MDT0001: reset Object Index mappings [ 9813.423399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9818.274731] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9820.828932] Lustre: 261532:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9827.355826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9833.606412] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9839.012359] LustreError: 261886:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9840.960192] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9841.157150] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9841.162381] Lustre: Skipped 1 previous similar message [ 9844.284116] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9844.303039] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9846.919272] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9854.257576] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9857.890082] Lustre: 263401:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9864.676965] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9864.683765] LustreError: 261665:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9864.685518] Lustre: Skipped 3 previous similar messages [ 9864.705511] LustreError: 261665:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9864.720251] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9864.768846] Lustre: server umount lustre-MDT0000 complete [ 9868.027352] LustreError: 260370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787893439 with bad export cookie 12118997243259084589 [ 9868.036466] LustreError: 260370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9868.037480] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9868.457877] Lustre: server umount lustre-MDT0001 complete [ 9882.690996] Lustre: server umount lustre-OST0000 complete [ 9896.344845] Lustre: server umount lustre-OST0001 complete [ 9912.228828] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 01:04:41 (1787893481) [ 9926.435491] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 9934.869344] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 9964.961602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9965.185680] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9965.202959] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9965.258700] Lustre: lustre-MDT0000: new disk, initializing [ 9965.322525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9965.341435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9971.338371] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9982.310039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9982.395345] Lustre: 267656:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9982.465444] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9982.472678] Lustre: Skipped 1 previous similar message [ 9982.540289] Lustre: lustre-MDT0001: new disk, initializing [ 9982.644621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9982.660042] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9987.946322] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9996.830819] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10002.457349] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10008.892214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10009.326609] Lustre: lustre-OST0000: new disk, initializing [10009.331303] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10009.338901] Lustre: 269626:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10010.764052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10010.773501] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10010.897615] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10015.845428] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10024.376085] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10029.037779] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10029.290929] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10034.892263] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10035.163517] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10041.826861] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10041.827040] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10041.831046] LustreError: Skipped 1 previous similar message [10041.839486] Lustre: Skipped 1 previous similar message [10041.842818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10043.415720] Lustre: server umount lustre-OST0000 complete [10049.504362] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10049.512698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10049.530587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10049.538611] Lustre: Skipped 1 previous similar message [10054.485344] Lustre: server umount lustre-MDT0000 complete [10057.188110] LustreError: 268552:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10057.204996] LustreError: 268552:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10057.883942] LustreError: 267648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787893629 with bad export cookie 12118997243259086682 [10057.894481] LustreError: 267648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10057.897511] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10058.136220] Lustre: server umount lustre-MDT0001 complete [10075.925461] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [10084.225822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10084.490137] LustreError: 272562:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10084.514718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10084.520057] Lustre: Skipped 2 previous similar messages [10088.619124] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10096.387369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10100.811190] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10103.524951] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10109.640686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10115.276082] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10121.819554] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10121.958298] Lustre: lustre-OST0001: new disk, initializing [10121.962765] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10121.969951] Lustre: 274869:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10123.783898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10123.792048] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10123.879380] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10126.958565] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10136.157451] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10140.081837] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10146.067201] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10146.096574] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10152.929166] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10152.939527] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10152.954079] Lustre: Skipped 1 previous similar message [10152.958679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10152.961441] Lustre: Skipped 2 previous similar messages [10158.627755] Lustre: server umount lustre-MDT0000 complete [10159.585265] LustreError: 273289:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10159.620746] LustreError: 273289:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [10162.474210] LustreError: 272542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787893733 with bad export cookie 12118997243259087564 [10162.479134] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10162.486206] LustreError: 272542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10162.883942] Lustre: server umount lustre-MDT0001 complete [10177.356206] Lustre: server umount lustre-OST0000 complete [10190.025940] Lustre: server umount lustre-OST0001 complete [10205.246104] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [10213.734684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10214.207893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10214.213961] Lustre: Skipped 3 previous similar messages [10217.379712] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10223.771104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10228.100409] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10230.396662] Lustre: 279075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10236.553587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10239.971427] LustreError: 279428:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10240.000817] LustreError: 279428:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10240.023360] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10241.981411] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10249.962796] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10255.357385] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10256.591435] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10265.492239] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10270.344902] Lustre: 280949:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10277.415534] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10277.529586] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10288.106289] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 01:10:57 (1787893857) [10306.016490] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10306.028324] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10306.047892] Lustre: Skipped 3 previous similar messages [10306.059539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10306.080585] Lustre: Skipped 4 previous similar messages [10310.005481] Lustre: server umount lustre-MDT0000 complete [10313.678189] LustreError: 278678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787893884 with bad export cookie 12118997243259094326 [10313.697521] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10320.032420] Lustre: server umount lustre-MDT0001 complete [10329.376333] Lustre: server umount lustre-OST0000 complete [10332.482895] Lustre: server umount lustre-OST0001 complete [10338.131875] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [10346.513381] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [10385.847351] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [10398.875922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10399.139877] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10399.206390] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10399.299836] Lustre: lustre-MDT0000: new disk, initializing [10399.427417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10403.951710] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10413.601565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10413.756608] Lustre: 285192:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10414.133163] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10418.573575] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10431.701425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10431.964683] Lustre: 286825:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10433.592799] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10437.284776] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10447.446959] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10447.630758] Lustre: 287696:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10449.110156] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10449.117915] Lustre: Skipped 1 previous similar message [10449.236359] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10453.407659] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10461.957447] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10465.067658] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10465.071572] Lustre: Skipped 1 previous similar message [10479.966566] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 01:14:09 (1787894049) [10485.223524] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10485.230573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10485.247341] Lustre: Skipped 7 previous similar messages [10485.262518] Lustre: Skipped 10 previous similar messages [10488.481691] Lustre: server umount lustre-MDT0000 complete [10490.342094] LustreError: 285203:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10490.356403] LustreError: 285203:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 14 previous similar messages [10492.518992] LustreError: 285185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787894063 with bad export cookie 12118997243259101760 [10492.522874] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10492.531628] LustreError: 285185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10492.894520] Lustre: server umount lustre-MDT0001 complete [10507.463892] Lustre: server umount lustre-OST0000 complete [10521.597642] Lustre: server umount lustre-OST0001 complete [10528.698282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10529.130388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10529.137143] Lustre: Skipped 7 previous similar messages [10533.042913] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10542.337407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10547.479411] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10555.995566] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10561.003328] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10566.412281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10571.549325] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10577.670195] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10594.467993] loop1: detected capacity change from 0 to 204800000 [10594.705880] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10594.714809] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10594.735739] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10600.251617] loop1: detected capacity change from 0 to 204800000 [10605.667200] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10605.741624] LustreError: 289933:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs13450: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10605.749821] LustreError: 289933:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs13450 [10605.753684] LustreError: 289933:0:(mgs_handler.c:517:mgs_target_reg()) fs13450-MDT0001: Error checking target index: rc = -13 [10605.759727] LustreError: fs13450-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10605.766700] LustreError: 293660:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10605.774845] LustreError: 293660:0:(tgt_mount.c:2129:server_put_super()) no obd fs13450-MDT0001 [10605.779162] LustreError: 293660:0:(tgt_mount.c:116:server_deregister_mount()) fs13450-MDT0001 not registered [10605.805411] Lustre: server umount fs13450-MDT0001 complete [10605.807800] LustreError: 293660:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10614.798546] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10614.922532] Lustre: Setting parameter fs13450-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs13450-MDT0001 [10614.977551] Lustre: srv-fs13450-MDT0001: No data found on store. Initialize space. [10614.981397] Lustre: Skipped 4 previous similar messages [10615.061518] Lustre: fs13450-MDT0001: new disk, initializing [10615.068287] Lustre: Skipped 3 previous similar messages [10615.212972] Lustre: srv-fs13450-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10615.223316] Lustre: 294127:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs13450-MDT0001-mdtlov: can't create new llog: rc = -115 [10615.231827] LustreError: 294127:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs13450-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10619.344328] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10625.744988] Lustre: server umount fs13450-MDT0001 complete [10638.104293] loop1: detected capacity change from 0 to 204800000 [10642.644455] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10642.901068] Lustre: 295694:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs13450-MDT0001-mdtlov: can't create new llog: rc = -115 [10642.912652] LustreError: 295694:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs13450-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10646.767519] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10650.604929] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10651.311518] Lustre: server umount fs13450-MDT0001 complete [10660.492385] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.44@tcp (stopping) [10660.836272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10660.850363] Lustre: Skipped 3 previous similar messages [10665.857870] Lustre: server umount lustre-MDT0000 complete [10674.711329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10674.802903] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10678.840924] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10680.301506] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10685.679768] loop1: detected capacity change from 0 to 204800000 [10689.998477] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10690.272791] Lustre: 298142:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs13450-MDT0001-mdtlov: can't create new llog: rc = -115 [10690.285851] LustreError: 298142:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs13450-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10693.912466] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10698.040800] Lustre: server umount fs13450-MDT0001 complete [10721.263872] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10721.274806] LustreError: Skipped 3 previous similar messages [10725.778175] Lustre: server umount lustre-OST0000 complete [10731.501682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10731.513766] Lustre: Skipped 7 previous similar messages [10743.263303] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10743.514596] Lustre: server umount lustre-MDT0000 complete [10746.849617] LustreError: 289945:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10746.858503] LustreError: 289945:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 99 previous similar messages [10747.132123] LustreError: 290693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787894318 with bad export cookie 12118997243259104441 [10747.142422] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10747.148078] LustreError: 290693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10747.480404] Lustre: server umount lustre-MDT0001 complete [10757.435062] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [10760.171023] Key type lgssc unregistered [10760.450635] LNet: 300490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10760.460500] LNetError: 300490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10760.486621] LNet: Removed LNI 192.168.203.144@tcp [10761.340157] Key type .llcrypt unregistered [10761.344135] Key type ._llcrypt unregistered [10781.025519] Key type ._llcrypt registered [10781.028556] Key type .llcrypt registered [10781.163871] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [10782.528876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10782.582223] alg: No test for adler32 (adler32-zlib) [10783.659468] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [10784.015667] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [10785.705326] Key type lgssc registered [10786.747940] Lustre: Echo OBD driver; http://www.lustre.org/ [10794.322586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10794.340946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10795.807594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10800.145711] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10806.245684] LustreError: 301762:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10806.266618] LustreError: 301762:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10807.644452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10808.025953] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10812.280535] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10819.298202] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10825.338929] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10833.306885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10833.612378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10839.952765] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10840.813952] LustreError: 303590:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10846.180497] LustreError: 303592:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10847.517633] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10850.440425] LustreError: 303592:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10855.526616] LustreError: 303592:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10855.554072] LustreError: 303592:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10863.591539] LustreError: 303590:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10863.626099] LustreError: 303590:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [10869.358550] loop1: detected capacity change from 0 to 204800000 [10869.616143] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10869.643962] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10869.656937] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10874.392538] loop1: detected capacity change from 0 to 204800000 [10879.998336] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10880.181656] Lustre: fs20713-OST0000: new disk, initializing [10880.186176] Lustre: srv-fs20713-OST0000: No data found on store. Initialize space. [10880.194200] Lustre: 305499:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10880.301434] Lustre: fs20713-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10881.125861] LustreError: 303592:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10881.145644] LustreError: 303592:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [10885.959261] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10894.251275] Lustre: server umount fs20713-OST0000 complete [10906.461365] loop1: detected capacity change from 0 to 204800000 [10911.786440] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10911.999261] Lustre: fs20713-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10914.791250] LustreError: 303590:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10914.812514] LustreError: 303590:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 19 previous similar messages [10918.788578] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10919.919655] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10926.781686] loop2: detected capacity change from 0 to 204800000 [10926.997081] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10927.014030] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10927.025361] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10932.132496] loop2: detected capacity change from 0 to 204800000 [10935.291761] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10937.595265] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10937.694632] LustreError: fs20713-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10937.702428] LustreError: fs20713-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10937.709245] LustreError: 308538:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10937.713301] LustreError: 308538:0:(tgt_mount.c:2129:server_put_super()) no obd fs20713-OST0001 [10937.717157] LustreError: 308538:0:(tgt_mount.c:116:server_deregister_mount()) fs20713-OST0001 not registered [10937.750512] Lustre: server umount fs20713-OST0001 complete [10937.753993] LustreError: 308538:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10947.128709] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10947.237892] Lustre: fs20713-OST0001: new disk, initializing [10947.243916] Lustre: srv-fs20713-OST0001: No data found on store. Initialize space. [10947.259848] Lustre: 309089:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10947.330435] Lustre: fs20713-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10953.175906] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10961.275190] Lustre: server umount fs20713-OST0000 complete [10970.203625] Lustre: server umount fs20713-OST0001 complete [10981.349821] LustreError: 303592:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10981.382071] LustreError: 303592:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 38 previous similar messages [10982.298663] loop1: detected capacity change from 0 to 204800000 [10988.349472] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10988.753416] Lustre: fs20713-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10996.513695] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11003.929414] loop2: detected capacity change from 0 to 204800000 [11010.138608] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11016.761945] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11025.335893] Lustre: server umount fs20713-OST0000 complete [11036.160283] Lustre: server umount fs20713-OST0001 complete [11050.979116] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11050.986773] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11051.017451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11053.033100] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11053.035774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11053.038516] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11053.069019] Lustre: Skipped 1 previous similar message [11054.628365] Lustre: server umount lustre-OST0000 complete [11059.168777] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11059.177273] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11059.190556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11063.271942] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11063.273544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11063.299154] Lustre: Skipped 1 previous similar message [11065.377394] Lustre: server umount lustre-MDT0000 complete [11069.285703] LustreError: 302504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787894640 with bad export cookie 120962080305728637 [11069.289397] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11069.304252] LustreError: 302504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11069.740760] Lustre: server umount lustre-MDT0001 complete [11080.584552] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [11083.293337] Key type lgssc unregistered [11083.615533] LNet: 315016:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11083.622301] LNetError: 315016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11084.651150] LNet: Removed LNI 192.168.203.144@tcp [11085.545036] Key type .llcrypt unregistered [11085.548086] Key type ._llcrypt unregistered [11119.359984] Key type ._llcrypt registered [11119.361780] Key type .llcrypt registered [11119.454594] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [11134.743643] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [11135.879282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11135.932426] alg: No test for adler32 (adler32-zlib) [11137.121198] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [11137.457728] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [11139.135329] Key type lgssc registered [11140.408311] Lustre: Echo OBD driver; http://www.lustre.org/ [11176.655041] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11176.675329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11177.979749] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11178.006185] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11178.107479] Lustre: lustre-MDT0000: new disk, initializing [11178.176089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11178.203916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11183.225065] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11194.869454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11194.998626] Lustre: 318879:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11195.050655] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11195.056410] Lustre: Skipped 1 previous similar message [11195.137685] Lustre: lustre-MDT0001: new disk, initializing [11195.243370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11195.293586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11195.311214] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11199.791356] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11209.610643] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11215.010539] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11222.100897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11222.431752] Lustre: lustre-OST0000: new disk, initializing [11222.434839] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11222.440831] Lustre: 320846:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11222.548519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11229.301375] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11231.284688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11231.294938] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11231.402904] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11238.833337] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11241.365170] Lustre: Mounted lustre-client - version 2.17.57_99_gb2219ab [11242.322743] Lustre: Unmounted lustre-client [11252.192205] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11252.193903] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11252.201324] LustreError: Skipped 1 previous similar message [11252.207682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11252.214856] Lustre: Skipped 1 previous similar message [11256.808433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11256.829339] Lustre: Skipped 1 previous similar message [11258.457740] Lustre: server umount lustre-OST0000 complete [11267.048376] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11267.051341] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11267.060435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11267.077924] Lustre: Skipped 2 previous similar messages [11269.196331] Lustre: server umount lustre-MDT0000 complete [11273.329622] LustreError: 318869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787894844 with bad export cookie 6322811360871144809 [11273.352536] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11273.358536] LustreError: 318869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11273.737495] Lustre: server umount lustre-MDT0001 complete [11283.753623] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [11286.520245] Key type lgssc unregistered [11293.152030] LNet: 1 local NIs in recovery (showing 1): 192.168.203.144@tcp [11294.053861] LNet: 323009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11294.070122] LNetError: 323009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11371.947164] LNet: Removed LNI 192.168.203.144@tcp [11373.665513] Key type .llcrypt unregistered [11373.669427] Key type ._llcrypt unregistered [11398.974186] Key type ._llcrypt registered [11398.978367] Key type .llcrypt registered [11399.122646] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [11416.385958] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [11417.611375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11417.692062] alg: No test for adler32 (adler32-zlib) [11418.788270] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [11419.029131] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [11420.703255] Key type lgssc registered [11421.896404] Lustre: Echo OBD driver; http://www.lustre.org/ [11454.236228] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 01:30:23 (1787895023) [11461.993757] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11462.015922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11463.348251] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11463.413528] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11463.532542] Lustre: lustre-MDT0000: new disk, initializing [11463.657733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11463.689266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11469.078486] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11480.173569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11480.276378] Lustre: 326237:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11480.353261] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11480.362786] Lustre: Skipped 1 previous similar message [11480.479180] Lustre: lustre-MDT0001: new disk, initializing [11480.608129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11480.643405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11480.656484] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11485.585230] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11496.681161] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11503.384312] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11511.211753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11511.637527] Lustre: lustre-OST0000: new disk, initializing [11511.640845] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11511.647770] Lustre: 328205:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11511.769291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11517.125650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11517.141219] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11517.293673] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11518.893465] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11528.036296] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11543.013214] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11543.014302] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11543.043364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11543.062334] Lustre: Skipped 1 previous similar message [11547.768395] Lustre: server umount lustre-OST0000 complete [11552.226524] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11552.233160] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11552.244349] Lustre: Skipped 1 previous similar message [11552.255499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11553.258767] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11553.283778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11553.287710] Lustre: Skipped 1 previous similar message [11558.485489] Lustre: server umount lustre-MDT0000 complete [11563.352468] LustreError: 326229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787895134 with bad export cookie 4553139621468605296 [11563.359284] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11563.363859] LustreError: 326229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11563.794336] Lustre: server umount lustre-MDT0001 complete [11578.191180] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [11581.916528] Key type lgssc unregistered [11582.196653] LNet: 330116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11582.208398] LNetError: 330116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11582.233551] LNet: Removed LNI 192.168.203.144@tcp [11583.327535] Key type .llcrypt unregistered [11583.333113] Key type ._llcrypt unregistered [11610.961025] Key type ._llcrypt registered [11610.967083] Key type .llcrypt registered [11611.179030] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [11612.888055] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11612.954698] alg: No test for adler32 (adler32-zlib) [11614.215930] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [11614.593263] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [11616.279220] Key type lgssc registered [11617.358686] Lustre: Echo OBD driver; http://www.lustre.org/ [11626.924551] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11626.982675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11628.664861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11633.472162] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11638.245699] LustreError: 331389:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11638.259995] LustreError: 331389:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11642.990128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11643.435644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11648.116709] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11655.926312] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11661.856722] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11669.816267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11670.128851] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11677.301229] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11687.593789] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11702.815816] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11704.810977] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11709.413024] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11709.419982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11709.431203] Lustre: Skipped 2 previous similar messages [11709.437201] Lustre: Skipped 1 previous similar message [11710.597434] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.44@tcp (stopping) [11710.603420] Lustre: Skipped 1 previous similar message [11714.536386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11714.545321] Lustre: Skipped 2 previous similar messages [11719.654737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11719.671964] Lustre: Skipped 3 previous similar messages [11723.231195] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11723.493577] Lustre: server umount lustre-MDT0000 complete [11724.772183] LustreError: 332388:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11724.787320] LustreError: 332388:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11725.929160] LustreError: 333229:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11727.511553] LustreError: 331370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787895298 with bad export cookie 842219876412227569 [11727.514792] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11727.526218] LustreError: 331370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11729.889131] LustreError: 331386:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11729.889914] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11729.936426] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11729.939563] Lustre: Skipped 1 previous similar message [11734.222986] Lustre: server umount lustre-MDT0001 complete [11738.456581] Lustre: server umount lustre-OST0000 complete [11746.375301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11752.282147] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11755.926482] Lustre: 335574:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.144@tcp [11758.673891] Lustre: server umount MGS complete [11767.312624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11767.890483] LustreError: 336146:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11767.909840] LustreError: 336146:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11767.963900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11772.216179] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11776.873205] LustreError: 336142:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11776.896652] LustreError: 336142:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11782.429983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11783.007205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11788.156437] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11797.792090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11798.203111] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11805.083516] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11814.256327] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11815.725257] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11821.202683] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11822.960572] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11829.216907] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11829.230965] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11829.238291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11829.251961] Lustre: Skipped 1 previous similar message [11829.265961] Lustre: Skipped 3 previous similar messages [11835.133079] Lustre: server umount lustre-OST0000 complete [11839.455857] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11839.462055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11839.478895] Lustre: Skipped 1 previous similar message [11845.485343] Lustre: server umount lustre-MDT0000 complete [11849.349640] LustreError: 336127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787895420 with bad export cookie 842219876412228409 [11849.367677] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11849.377969] LustreError: 336127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11849.720225] Lustre: server umount lustre-MDT0001 complete [11861.922389] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [11865.738995] Key type lgssc unregistered [11866.142953] LNet: 339590:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11866.176081] LNetError: 339590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11866.203777] LNet: Removed LNI 192.168.203.144@tcp [11867.042873] Key type .llcrypt unregistered [11867.045115] Key type ._llcrypt unregistered [11888.707176] Key type ._llcrypt registered [11888.709168] Key type .llcrypt registered [11888.809458] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [11907.954495] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [11909.304402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11909.499722] alg: No test for adler32 (adler32-zlib) [11910.702409] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [11910.970117] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [11912.696093] Key type lgssc registered [11913.935764] Lustre: Echo OBD driver; http://www.lustre.org/ [11952.318399] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11952.363285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11953.654282] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11953.704373] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11953.807710] Lustre: lustre-MDT0000: new disk, initializing [11953.950154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11953.988516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11959.288413] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11970.773916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11970.934436] Lustre: 342751:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11971.017254] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11971.023604] Lustre: Skipped 1 previous similar message [11971.105550] Lustre: lustre-MDT0001: new disk, initializing [11971.220792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11971.252702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11971.265243] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11976.685428] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11986.733630] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11992.541960] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11999.712235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12000.043672] Lustre: lustre-OST0000: new disk, initializing [12000.048251] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12000.056738] Lustre: 344722:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12000.150928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12005.580616] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12005.940297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12005.951784] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12006.030344] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12014.817575] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12020.708121] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12021.022983] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12026.550804] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12026.759337] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12031.462347] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12031.464635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12031.478383] Lustre: Skipped 1 previous similar message [12031.492419] Lustre: Skipped 1 previous similar message [12034.636985] Lustre: server umount lustre-OST0000 complete [12041.705109] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12041.725051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12041.725670] Lustre: Skipped 1 previous similar message [12041.750772] Lustre: Skipped 1 previous similar message [12044.682915] Lustre: server umount lustre-MDT0000 complete [12048.153850] LustreError: 344721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787895619 with bad export cookie 7678486117892430843 [12048.167916] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12048.181890] LustreError: 344721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12048.662439] Lustre: server umount lustre-MDT0001 complete [12058.035746] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 01:40:27 (1787895627) [12076.060325] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [12085.756717] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [12122.596395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12122.833315] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12122.854580] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12122.936325] Lustre: lustre-MDT0000: new disk, initializing [12123.002526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12123.023774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12127.251310] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12139.394679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12139.631357] Lustre: 349483:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12139.728566] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12139.733897] Lustre: Skipped 1 previous similar message [12139.854791] Lustre: lustre-MDT0001: new disk, initializing [12140.041597] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12140.080427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12140.098409] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12145.680035] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12156.896763] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12163.099660] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12169.190057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12169.517025] Lustre: lustre-OST0000: new disk, initializing [12169.520294] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12169.528954] Lustre: 351452:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12169.601572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12175.934840] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12176.425963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12176.452137] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12176.574310] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12187.907901] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12226.020818] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12226.039025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12231.143785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12231.149025] Lustre: Skipped 2 previous similar messages [12236.266960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12236.274317] Lustre: Skipped 1 previous similar message [12239.327360] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12239.559789] Lustre: server umount lustre-OST0000 complete [12246.502038] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12246.515743] Lustre: Skipped 1 previous similar message [12246.522155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12246.530706] Lustre: Skipped 1 previous similar message [12258.271199] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12258.553636] Lustre: server umount lustre-MDT0000 complete [12261.858205] LustreError: 352270:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12261.908724] LustreError: 352270:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [12263.619701] LustreError: 349473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787895834 with bad export cookie 7678486117892431725 [12263.620274] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12263.631630] LustreError: 349473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12264.063564] Lustre: server umount lustre-MDT0001 complete [12279.285973] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12282.568023] Key type lgssc unregistered [12282.839455] LNet: 353630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12282.844442] LNetError: 353630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12282.857293] LNet: Removed LNI 192.168.203.144@tcp [12283.668401] Key type .llcrypt unregistered [12283.675141] Key type ._llcrypt unregistered [12380.353850] Key type ._llcrypt registered [12380.356107] Key type .llcrypt registered [12380.493872] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12380.808555] Key type .llcrypt unregistered [12380.814694] Key type ._llcrypt unregistered [12466.901233] Key type ._llcrypt registered [12466.903840] Key type .llcrypt registered [12467.127634] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12467.639402] Key type .llcrypt unregistered [12467.643644] Key type ._llcrypt unregistered [12503.534326] Key type ._llcrypt registered [12503.553708] Key type .llcrypt registered [12503.695272] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12504.109317] Key type .llcrypt unregistered [12504.111882] Key type ._llcrypt unregistered [12533.210357] Key type ._llcrypt registered [12533.214308] Key type .llcrypt registered [12533.329940] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12533.648697] Key type .llcrypt unregistered [12533.651405] Key type ._llcrypt unregistered [12610.735925] Key type ._llcrypt registered [12610.738439] Key type .llcrypt registered [12610.903422] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12611.385538] Key type .llcrypt unregistered [12611.392254] Key type ._llcrypt unregistered [12641.169155] Key type ._llcrypt registered [12641.171841] Key type .llcrypt registered [12641.323929] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12641.791311] Key type .llcrypt unregistered [12641.799964] Key type ._llcrypt unregistered [12677.004092] Key type ._llcrypt registered [12677.018740] Key type .llcrypt registered [12677.119843] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12677.414213] Key type .llcrypt unregistered [12677.417901] Key type ._llcrypt unregistered [12716.747517] Key type ._llcrypt registered [12716.751567] Key type .llcrypt registered [12716.896439] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12717.132136] Key type .llcrypt unregistered [12717.134992] Key type ._llcrypt unregistered [12746.499180] Key type ._llcrypt registered [12746.501860] Key type .llcrypt registered [12746.627384] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12746.933210] Key type .llcrypt unregistered [12746.935621] Key type ._llcrypt unregistered [12777.927367] Key type ._llcrypt registered [12777.929279] Key type .llcrypt registered [12778.058125] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12778.380474] Key type .llcrypt unregistered [12778.382974] Key type ._llcrypt unregistered [12855.131623] Key type ._llcrypt registered [12855.139024] Key type .llcrypt registered [12855.328474] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12855.805568] Key type .llcrypt unregistered [12855.813686] Key type ._llcrypt unregistered [12887.652996] Key type ._llcrypt registered [12887.655736] Key type .llcrypt registered [12887.755373] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12888.029293] Key type .llcrypt unregistered [12888.035266] Key type ._llcrypt unregistered [12917.591635] Key type ._llcrypt registered [12917.593017] Key type .llcrypt registered [12917.652442] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12917.963275] Key type .llcrypt unregistered [12917.966123] Key type ._llcrypt unregistered [12977.223722] Key type ._llcrypt registered [12977.225716] Key type .llcrypt registered [12977.324555] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [12994.240890] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [12995.564909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12995.576985] alg: No test for adler32 (adler32-zlib) [12996.751464] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [12997.251494] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [12998.993395] Key type lgssc registered [13000.813586] Lustre: Echo OBD driver; http://www.lustre.org/ [13042.454675] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13042.481674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13043.764686] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13043.809973] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13043.923656] Lustre: tst250fs-MDT0000: new disk, initializing [13044.076429] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13044.109891] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13049.050788] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [13060.066586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13060.214298] Lustre: 374003:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13060.333725] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13060.352208] Lustre: Skipped 1 previous similar message [13060.467374] Lustre: tst250fs-MDT0001: new disk, initializing [13060.487296] Lustre: tst250fs-MDT0001: Not available for connect from 0@lo (not set up) [13060.593393] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13060.665605] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13060.696144] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13065.921246] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [13071.291031] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13078.603897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13079.099486] Lustre: tst250fs-OST0000: new disk, initializing [13079.109215] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13079.119947] Lustre: 375636:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13079.263396] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13084.291923] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13084.317303] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13084.477145] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13086.467884] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [13096.499457] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13096.690978] Lustre: tst250fs-OST0001: new disk, initializing [13096.695486] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13096.701845] Lustre: 376510:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13096.789098] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13104.290332] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [13106.267530] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13106.281698] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13106.415136] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13110.578568] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13112.703195] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [13112.708776] Lustre: Skipped 1 previous similar message [13137.380924] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13137.381600] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13137.418932] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13137.440636] Lustre: Skipped 3 previous similar messages [13139.598493] Lustre: server umount tst250fs-MDT0000 complete [13143.907531] LustreError: 374799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787896715 with bad export cookie 1772806089124099532 [13143.908542] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13143.913049] LustreError: 374799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13144.601490] Lustre: server umount tst250fs-MDT0001 complete [13160.319455] Lustre: server umount tst250fs-OST0000 complete [13163.495965] Lustre: 372228:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787896718/real 1787896718] req@ffff99348c9b1a40 x1874745431758720/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787896734 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13163.539457] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13163.554380] Lustre: Skipped 3 previous similar messages [13164.000137] Lustre: 372226:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787896719/real 1787896719] req@ffff993586789680 x1874745431758976/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787896735 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13164.598737] Lustre: server umount tst250fs-OST0001 complete [13187.791295] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [13198.841961] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [13233.570686] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 02:00:03 (1787896803) [13235.260377] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13255.613569] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [13267.510061] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [13303.041261] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13006 sec ========= 02:01:12 (1787896872) [13304.962437] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 02:01:14 (1787896874) === [13309.497680] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 02:01:18 (1787896878) === [13351.258945] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [13354.068269] Key type lgssc unregistered [13354.379879] LNet: 385675:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13354.389515] LNetError: 385675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13354.423352] LNet: Removed LNI 192.168.203.144@tcp [13355.413168] Key type .llcrypt unregistered [13355.415271] Key type ._llcrypt unregistered