[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 474754330 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003293] x2apic enabled [ 0.004011] Switched APIC routing to physical x2apic. [ 0.006014] kvm-guest: setup PV IPIs [ 0.008936] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009032] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010017] pid_max: default: 32768 minimum: 301 [ 0.011155] LSM: Security Framework initializing [ 0.012083] Yama: becoming mindful. [ 0.013063] SELinux: Initializing. [ 0.014097] *** VALIDATE selinux *** [ 0.024158] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029128] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.031139] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032138] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034027] *** VALIDATE tmpfs *** [ 0.036041] *** VALIDATE proc *** [ 0.038141] *** VALIDATE cgroup *** [ 0.039015] *** VALIDATE cgroup2 *** [ 0.041268] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.043058] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.044012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.045043] Spectre V2 : User space: Vulnerable [ 0.046010] Speculative Store Bypass: Vulnerable [ 0.049412] debug: unmapping init [mem 0xffffffffaf659000-0xffffffffaf660fff] [ 0.052188] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.053829] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.054034] ... version: 2 [ 0.055025] ... bit width: 48 [ 0.056023] ... generic registers: 4 [ 0.057021] ... value mask: 0000ffffffffffff [ 0.058022] ... max period: 00007fffffffffff [ 0.059027] ... fixed-purpose events: 3 [ 0.060020] ... event mask: 000000070000000f [ 0.062322] rcu: Hierarchical SRCU implementation. [ 0.064844] smp: Bringing up secondary CPUs ... [ 0.065723] x86: Booting SMP configuration: [ 0.066046] .... node #0, CPUs: #1 #2 #3 [ 0.070018] smp: Brought up 1 node, 4 CPUs [ 0.072029] smpboot: Max logical packages: 1 [ 0.073037] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.240023] node 0 deferred pages initialised in 165ms [ 0.243429] devtmpfs: initialized [ 0.244270] x86/mm: Memory block size: 128MB [ 0.246836] gcov: version magic: 0x41383552 [ 0.248366] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.252087] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.256377] pinctrl core: initialized pinctrl subsystem [ 0.258170] [ 0.258632] ************************************************************* [ 0.261013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.263011] ** ** [ 0.265013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.267015] ** ** [ 0.269011] ** This means that this kernel is built to expose internal ** [ 0.272015] ** IOMMU data structures, which may compromise security on ** [ 0.274012] ** your system. ** [ 0.277015] ** ** [ 0.280015] ** If you see this message and you are not debugging the ** [ 0.283015] ** kernel, report this immediately to your vendor! ** [ 0.285012] ** ** [ 0.288014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.291025] ************************************************************* [ 0.295757] NET: Registered protocol family 16 [ 0.298469] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.303066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.306060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.309106] cpuidle: using governor menu [ 0.310976] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.311684] PCI: Using configuration type 1 for base access [ 0.313135] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.323150] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.326022] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.329153] cryptd: max_cpu_qlen set to 1000 [ 0.334290] ACPI: Added _OSI(Module Device) [ 0.337006] ACPI: Added _OSI(Processor Device) [ 0.339020] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.340013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.345082] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.351634] ACPI: Interpreter enabled [ 0.353070] ACPI: PM: (supports S0 S3 S4 S5) [ 0.355015] ACPI: Using IOAPIC for interrupt routing [ 0.357101] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.360425] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.370495] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.373044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.376017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.379076] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.385369] acpiphp: Slot [2] registered [ 0.386185] acpiphp: Slot [5] registered [ 0.388145] acpiphp: Slot [6] registered [ 0.390111] acpiphp: Slot [7] registered [ 0.391145] acpiphp: Slot [8] registered [ 0.393127] acpiphp: Slot [9] registered [ 0.395118] acpiphp: Slot [10] registered [ 0.396123] acpiphp: Slot [3] registered [ 0.398108] acpiphp: Slot [4] registered [ 0.400102] acpiphp: Slot [11] registered [ 0.401101] acpiphp: Slot [12] registered [ 0.403126] acpiphp: Slot [13] registered [ 0.404095] acpiphp: Slot [14] registered [ 0.406181] acpiphp: Slot [15] registered [ 0.407084] acpiphp: Slot [16] registered [ 0.409092] acpiphp: Slot [17] registered [ 0.410097] acpiphp: Slot [18] registered [ 0.412116] acpiphp: Slot [19] registered [ 0.413088] acpiphp: Slot [20] registered [ 0.415128] acpiphp: Slot [21] registered [ 0.416100] acpiphp: Slot [22] registered [ 0.418100] acpiphp: Slot [23] registered [ 0.420121] acpiphp: Slot [24] registered [ 0.421178] acpiphp: Slot [25] registered [ 0.423100] acpiphp: Slot [26] registered [ 0.425133] acpiphp: Slot [27] registered [ 0.427105] acpiphp: Slot [28] registered [ 0.428110] acpiphp: Slot [29] registered [ 0.430086] acpiphp: Slot [30] registered [ 0.431098] acpiphp: Slot [31] registered [ 0.432056] PCI host bridge to bus 0000:00 [ 0.434022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.437026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.440025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.442023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.445032] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.448035] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.450197] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.453092] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.457135] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.467831] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.473107] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.476021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.479026] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.482020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.485351] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.487788] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.491047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.494844] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.500015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.514960] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.520014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.527134] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.535016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.543014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.562015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.575921] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.584012] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.590015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.607016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.620204] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.632016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.643016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.666018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.677829] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.688015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.700073] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.720015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.736000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.747016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.756024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.775017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.786874] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.798015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.804015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.829019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.839975] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.842296] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.844379] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.847422] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.850179] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.854252] iommu: Default domain type: Passthrough [ 0.856391] SCSI subsystem initialized [ 0.858135] ACPI: bus type USB registered [ 0.859090] usbcore: registered new interface driver usbfs [ 0.861104] usbcore: registered new interface driver hub [ 0.863076] usbcore: registered new device driver usb [ 0.865204] pps_core: LinuxPPS API ver. 1 registered [ 0.867011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.870050] PTP clock support registered [ 0.872086] EDAC MC: Ver: 3.0.0 [ 0.874110] PCI: Using ACPI for IRQ routing [ 0.876909] NetLabel: Initializing [ 0.878010] NetLabel: domain hash size = 128 [ 0.879009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.881097] NetLabel: unlabeled traffic allowed by default [ 0.883157] vgaarb: loaded [ 0.884248] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.886015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.892335] clocksource: Switched to clocksource kvm-clock [ 0.998917] VFS: Disk quotas dquot_6.6.0 [ 1.000419] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.002574] *** VALIDATE ramfs *** [ 1.003744] *** VALIDATE hugetlbfs *** [ 1.005139] pnp: PnP ACPI init [ 1.007159] pnp: PnP ACPI: found 6 devices [ 1.029412] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.033236] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.035672] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.038308] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.041169] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.043795] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.046938] NET: Registered protocol family 2 [ 1.049709] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.054641] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.057618] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.062891] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.066431] TCP: Hash tables configured (established 65536 bind 65536) [ 1.069059] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.071740] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.074399] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.076892] NET: Registered protocol family 1 [ 1.081769] RPC: Registered named UNIX socket transport module. [ 1.083791] RPC: Registered udp transport module. [ 1.085511] RPC: Registered tcp transport module. [ 1.086913] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.088843] NET: Registered protocol family 44 [ 1.090920] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.093321] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.095738] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.098232] PCI: CLS 0 bytes, default 64 [ 1.099860] Unpacking initramfs... [ 2.485219] debug: unmapping init [mem 0xffff8905bcc54000-0xffff8905bffbffff] [ 2.488727] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.491716] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.495182] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.994724] Initialise system trusted keyrings [ 2.996445] Key type blacklist registered [ 2.998276] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.009857] zbud: loaded [ 3.013903] *** VALIDATE nfs *** [ 3.015327] *** VALIDATE nfs4 *** [ 3.017057] pstore: using deflate compression [ 3.021480] Platform Keyring initialized [ 3.122328] NET: Registered protocol family 38 [ 3.124368] Key type asymmetric registered [ 3.126163] Asymmetric key parser 'x509' registered [ 3.128397] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.131841] io scheduler mq-deadline registered [ 3.134117] io scheduler kyber registered [ 3.136252] io scheduler bfq registered [ 3.138412] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.141660] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.144775] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.148109] ACPI: Power Button [PWRF] [ 3.154089] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.159537] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.173502] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.182942] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.204558] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.233010] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.263053] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.268621] Non-volatile memory driver v1.3 [ 3.270443] Linux agpgart interface v0.103 [ 3.302782] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.306395] vda: detected capacity change from 0 to 74702848 [ 3.319964] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.323331] vdb: detected capacity change from 0 to 1073741824 [ 3.336310] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.339616] vdc: detected capacity change from 0 to 2621440000 [ 3.353779] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.357090] vdd: detected capacity change from 0 to 2621440000 [ 3.372267] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.375723] vde: detected capacity change from 0 to 4294967296 [ 3.390263] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.393712] vdf: detected capacity change from 0 to 4294967296 [ 3.400449] libphy: Fixed MDIO Bus: probed [ 3.406086] usbcore: registered new interface driver usbserial_generic [ 3.408388] usbserial: USB Serial support registered for generic [ 3.410823] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.415116] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.416954] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.419685] mousedev: PS/2 mouse device common for all mice [ 3.422875] rtc_cmos 00:05: RTC can wake from S4 [ 3.425824] rtc_cmos 00:05: registered as rtc0 [ 3.427908] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.428361] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.432636] intel_pstate: CPU model not supported [ 3.435508] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.436172] hid: raw HID events driver (C) Jiri Kosina [ 3.440368] usbcore: registered new interface driver usbhid [ 3.440456] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.441895] usbhid: USB HID core driver [ 3.445855] drop_monitor: Initializing network drop monitor service [ 3.448447] Initializing XFRM netlink socket [ 3.450147] NET: Registered protocol family 10 [ 3.452596] Segment Routing with IPv6 [ 3.454236] NET: Registered protocol family 17 [ 3.456077] mpls_gso: MPLS GSO support [ 3.461404] RAS: Correctable Errors collector initialized. [ 3.463858] AVX version of gcm_enc/dec engaged. [ 3.465697] AES CTR mode by8 optimization enabled [ 3.542489] sched_clock: Marking stable (3542459252, 0)->(4490933058, -948473806) [ 3.547066] registered taskstats version 1 [ 3.549393] Loading compiled-in X.509 certificates [ 3.551898] zswap: loaded using pool lzo/zbud [ 3.577731] Key type big_key registered [ 3.591650] Key type encrypted registered [ 3.593469] ima: No TPM chip found, activating TPM-bypass! [ 3.595384] ima: Allocated hash algorithm: sha1 [ 3.597140] ima: No architecture policies found [ 3.598977] evm: Initialising EVM extended attributes: [ 3.601010] evm: security.selinux [ 3.602454] evm: security.ima [ 3.603519] evm: security.capability [ 3.604837] evm: HMAC attrs: 0x1 [ 3.607312] rtc_cmos 00:05: setting system clock to 2026-08-31 08:20:32 UTC (1788164432) [ 3.612928] debug: unmapping init [mem 0xffffffffb0603000-0xffffffffb07fffff] [ 3.616055] debug: unmapping init [mem 0xffffffffaf382000-0xffffffffaf658fff] [ 3.629161] Write protecting the kernel read-only data: 28672k [ 3.632269] debug: unmapping init [mem 0xffffffffada03000-0xffffffffadbfffff] [ 3.635240] debug: unmapping init [mem 0xffffffffae314000-0xffffffffae3fffff] [ 3.669826] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.678702] systemd[1]: Detected virtualization kvm. [ 3.680975] systemd[1]: Detected architecture x86-64. [ 3.683096] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.711342] systemd[1]: No hostname configured. [ 3.712992] systemd[1]: Set hostname to . [ 3.715081] random: systemd: uninitialized urandom read (16 bytes read) [ 3.717400] systemd[1]: Initializing machine ID from random generator. [ 3.769252] random: ln: uninitialized urandom read (6 bytes read) [ 3.847344] random: systemd: uninitialized urandom read (16 bytes read) [ 3.857914] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.865423] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.871528] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Paths. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.490605] device-mapper: uevent: version 1.0.3 [ 4.493201] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Startin[ 5.126468] virtio_net virtio0 ens2: renamed from eth0 g dracut initqueue hook... [ 5.159046] random: fast init done [ 5.189751] scsi host0: ata_piix [ 5.200082] scsi host1: ata_piix [ 5.201875] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.204717] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.386211] dracut-initqueue[586]: RTNETLINK answers: File exists [ 10.109216] random: crng init done [ 10.110737] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.428981] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.512595] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.782834] SELinux: Disabled at runtime. [ 11.846437] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.854820] systemd[1]: Detected virtualization kvm. [ 11.856287] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.329250] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.332884] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.338121] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.342846] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.346802] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.354887] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.358426] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. Mounting Kernel Debug File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ 12.513345] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ 12.748570] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /mnt. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.064431] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.111161] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.201993] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.214365] EDAC sbridge: Ver: 1.1.2 [ 14.825950] Key type dns_resolver registered [ 15.129643] NFS: Registering the id_resolver key type [ 15.131897] Key type id_resolver registered [ 15.134053] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Network Manager... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg345-server login: [ 40.087534] libcfs: loading out-of-tree module taints kernel. [ 40.110241] Key type ._llcrypt registered [ 40.112457] Key type .llcrypt registered [ 40.168356] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 49.295652] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 50.050747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 50.058589] alg: No test for adler32 (adler32-zlib) [ 51.096909] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 51.466959] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 53.103164] Key type lgssc registered [ 54.047222] Lustre: Echo OBD driver; http://www.lustre.org/ [ 69.725086] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 105.277634] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 116.615267] hrtimer: interrupt took 7162305 ns [ 118.919581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 118.951809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 120.248219] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 120.296708] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 120.405303] Lustre: lustre-MDT0000: new disk, initializing [ 120.481161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 120.508843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 125.880608] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 142.418282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 142.635424] Lustre: 6481:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 142.689260] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 142.698208] Lustre: Skipped 1 previous similar message [ 142.838769] Lustre: lustre-MDT0001: new disk, initializing [ 142.990661] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 143.047660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 143.060927] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 148.939379] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 153.451537] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 163.203352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 163.638018] Lustre: lustre-OST0000: new disk, initializing [ 163.642702] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 163.647401] Lustre: 8422:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 163.701827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 165.074619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 165.107665] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 165.185624] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 170.715480] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 186.766766] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 186.914835] Lustre: lustre-OST0001: new disk, initializing [ 186.919384] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 186.924359] Lustre: 9493:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 186.994204] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 193.648888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 193.662725] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 193.784243] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 193.931718] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 207.215641] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 216.618666] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 222.621693] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing check_logdir /tmp/testlogs/ [ 229.340744] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing yml_node [ 234.611990] Lustre: DEBUG MARKER: Client: 2.17.56.54 [ 237.510583] Lustre: DEBUG MARKER: MDS: 2.17.56.54 [ 240.105574] Lustre: DEBUG MARKER: OSS: 2.17.56.54 [ 241.695523] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 31 04:24:28 EDT 2026 [ 257.628826] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 258.925184] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 275.936689] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 275.942692] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 275.943033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 275.949742] Lustre: Skipped 1 previous similar message [ 280.068362] Lustre: server umount lustre-MDT0000 complete [ 283.529970] LustreError: 10310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788164712 with bad export cookie 18120253457493774289 [ 283.530878] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 283.545616] LustreError: 10310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 286.175800] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 286.177030] LustreError: 6489:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 286.178643] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 286.178653] Lustre: Skipped 3 previous similar messages [ 286.214793] Lustre: Skipped 3 previous similar messages [ 286.244212] LustreError: 6489:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 288.223915] LustreError: 6490:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 288.224068] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 288.239493] Lustre: Skipped 2 previous similar messages [ 289.942953] Lustre: server umount lustre-MDT0001 complete [ 294.567811] Lustre: server umount lustre-OST0000 complete [ 298.870721] Lustre: server umount lustre-OST0001 complete [ 305.078422] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 315.257468] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 350.753672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 350.976694] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 351.012901] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 351.096354] Lustre: lustre-MDT0000: new disk, initializing [ 351.156880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 351.175136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 355.586914] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 365.049490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 365.150860] Lustre: 15047:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 365.197504] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 365.201892] Lustre: Skipped 1 previous similar message [ 365.276323] Lustre: lustre-MDT0001: new disk, initializing [ 365.375350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 365.442104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 365.455827] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 369.536676] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 378.781261] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 384.697868] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 391.352827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 391.688063] Lustre: lustre-OST0000: new disk, initializing [ 391.693913] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 391.699844] Lustre: 17011:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 391.806960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 393.026263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 393.035446] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 393.137107] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 397.699856] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 407.004966] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 411.515924] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 411.723592] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 416.301558] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 416.540438] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 418.787941] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 418.801806] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 418.802762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 418.814020] Lustre: Skipped 1 previous similar message [ 423.906788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 423.918062] Lustre: Skipped 2 previous similar messages [ 424.529673] Lustre: server umount lustre-OST0000 complete [ 429.035839] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 434.146655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 434.156552] Lustre: Skipped 3 previous similar messages [ 442.336298] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 442.587303] Lustre: server umount lustre-MDT0000 complete [ 444.387325] LustreError: 15057:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 444.415121] LustreError: 15057:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 446.339340] LustreError: 15039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788164875 with bad export cookie 18120253457493775822 [ 446.340977] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 446.346531] LustreError: 15039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 446.678380] Lustre: server umount lustre-MDT0001 complete [ 452.670790] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 04:27:59 (1788164879) [ 472.872840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 473.590974] LustreError: 19889:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 473.682082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 478.692714] LustreError: 19890:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 479.698626] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 488.933764] LustreError: 19889:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 488.984283] LustreError: 19889:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 489.809567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 495.365482] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 504.021927] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 510.606739] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 512.911891] Lustre: Failing over lustre-MDT0000 [ 512.927848] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 512.942688] Lustre: Skipped 1 previous similar message [ 512.954972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 512.966706] Lustre: Skipped 3 previous similar messages [ 513.489939] Lustre: server umount lustre-MDT0000 complete [ 516.066527] LustreError: 20617:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 516.095753] LustreError: 20617:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 531.423143] Lustre: 3617:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788164944/real 1788164944] req@ffff89063f4c9c00 x1875026357941888/t0(0) o400->MGC192.168.203.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788164960 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 531.467603] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 533.878324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 542.695684] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xfb781283adf5db54 [ 542.707748] Lustre: MGC192.168.203.145@tcp: Connection restored to 0@lo (at 0@lo) [ 542.907420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 542.912134] Lustre: Skipped 1 previous similar message [ 546.902067] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 548.323762] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 548.329592] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 548.363415] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 557.254589] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 563.685701] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 563.694865] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 563.717814] Lustre: Skipped 1 previous similar message [ 563.735088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 567.600818] Lustre: server umount lustre-MDT0000 complete [ 570.854507] LustreError: 19886:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 570.890859] LustreError: 19886:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 571.799119] LustreError: 19870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788165000 with bad export cookie 18120253457493777236 [ 571.804968] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 571.816533] LustreError: 19870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 572.105238] Lustre: server umount lustre-MDT0001 complete [ 581.492932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 586.711301] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 595.918500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 600.796932] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 608.041025] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 613.781284] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 616.091742] Lustre: Failing over lustre-MDT0000 [ 616.383202] Lustre: server umount lustre-MDT0000 complete [ 616.931314] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 616.931766] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 616.961549] Lustre: Skipped 2 previous similar messages [ 635.258502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 635.389943] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 635.672049] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 640.575275] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 640.996561] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 641.005402] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 641.025771] Lustre: Skipped 1 previous similar message [ 641.062426] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 651.966180] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 660.449359] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 660.472459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 660.474848] Lustre: Skipped 2 previous similar messages [ 663.510818] Lustre: server umount lustre-MDT0000 complete [ 665.568261] LustreError: 23227:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 665.579576] LustreError: 23227:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 667.266123] LustreError: 23208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788165096 with bad export cookie 18120253457493778174 [ 667.279668] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 667.284227] LustreError: 23208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 667.653346] Lustre: server umount lustre-MDT0001 complete [ 676.755700] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 04:31:43 (1788165103) [ 692.653465] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 702.801807] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 745.662657] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 758.253308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 758.448560] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 758.489788] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 758.652617] Lustre: lustre-MDT0000: new disk, initializing [ 758.753286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 758.765158] Lustre: Skipped 3 previous similar messages [ 758.776712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 762.219552] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 772.435769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 772.652549] Lustre: 29601:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 772.880191] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 776.127838] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 785.465143] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 792.588594] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 800.694461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 801.308605] Lustre: 31572:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 802.408245] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 802.470425] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 802.495713] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 802.506093] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 810.982083] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 823.833896] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 831.651647] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 853.984563] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 853.991293] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 853.994751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 853.999796] Lustre: Skipped 2 previous similar messages [ 854.015840] Lustre: Skipped 1 previous similar message [ 868.320811] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 868.599292] Lustre: server umount lustre-OST0000 complete [ 879.359725] Lustre: server umount lustre-MDT0000 complete [ 879.585889] LustreError: 29614:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 879.611341] LustreError: 29614:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 883.810497] LustreError: 30407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788165312 with bad export cookie 18120253457493778580 [ 883.819573] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 883.832485] LustreError: 30407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 884.273748] Lustre: server umount lustre-MDT0001 complete [ 897.456920] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 900.306324] Key type lgssc unregistered [ 900.637884] LNet: 33554:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 900.645922] LNetError: 33554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 901.686675] LNet: Removed LNI 192.168.203.145@tcp [ 902.792233] Key type .llcrypt unregistered [ 902.795524] Key type ._llcrypt unregistered [ 927.809255] Key type ._llcrypt registered [ 927.813947] Key type .llcrypt registered [ 927.928493] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 929.348198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 929.404630] alg: No test for adler32 (adler32-zlib) [ 930.400625] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 930.604887] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 932.280684] Key type lgssc registered [ 933.380515] Lustre: Echo OBD driver; http://www.lustre.org/ [ 941.281581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 941.301543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 942.806569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 947.546531] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 952.291791] LustreError: 34974:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 952.305982] LustreError: 34974:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 956.688921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 957.111139] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 961.788076] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 969.635408] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 976.695939] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 984.603564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 985.163130] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 992.235280] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1001.597243] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1001.858157] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1014.544534] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 04:37:21 (1788165441) [ 1016.220619] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1017.797536] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1023.934937] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 04:37:30 (1788165450) [ 1032.034401] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 04:37:38 (1788165458) [ 1032.861301] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1042.111805] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1042.116243] Lustre: Skipped 2 previous similar messages [ 1044.492220] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 04:37:50 (1788165470) [ 1046.480678] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1046.487816] Lustre: Skipped 2 previous similar messages [ 1054.868617] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1054.878064] Lustre: Skipped 4 previous similar messages [ 1072.473230] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 04:38:18 (1788165498) [ 1075.938550] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1079.529197] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1088.564089] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 04:38:35 (1788165515) [ 1090.280498] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1090.287301] Lustre: Skipped 6 previous similar messages [ 1102.676217] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 04:38:49 (1788165529) [ 1119.713032] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1119.714556] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1119.724884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1119.741017] Lustre: Skipped 1 previous similar message [ 1123.417441] Lustre: server umount lustre-OST0000 complete [ 1129.953736] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1129.970799] Lustre: Skipped 1 previous similar message [ 1129.975378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1133.650705] Lustre: server umount lustre-MDT0000 complete [ 1135.073773] LustreError: 35701:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1135.095942] LustreError: 35701:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1137.227622] LustreError: 34954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788165566 with bad export cookie 12675091741126028724 [ 1137.236295] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1137.243854] LustreError: 34954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1137.578551] Lustre: server umount lustre-MDT0001 complete [ 1147.667729] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 1150.109637] Key type lgssc unregistered [ 1150.405283] LNet: 42831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1150.410675] LNetError: 42831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1151.473984] LNet: Removed LNI 192.168.203.145@tcp [ 1152.276269] Key type .llcrypt unregistered [ 1152.278521] Key type ._llcrypt unregistered [ 1169.706871] Key type ._llcrypt registered [ 1169.708957] Key type .llcrypt registered [ 1169.807112] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 1187.437289] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 1188.874368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1188.923214] alg: No test for adler32 (adler32-zlib) [ 1189.883213] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 1190.107360] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 1191.785613] Key type lgssc registered [ 1193.224643] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1229.846141] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1229.879805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1231.121560] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1231.155720] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1231.240474] Lustre: lustre-MDT0000: new disk, initializing [ 1231.319542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1231.347157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1235.966779] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1246.697753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1246.832450] Lustre: 45985:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1246.937665] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1246.948347] Lustre: Skipped 1 previous similar message [ 1247.047029] Lustre: lustre-MDT0001: new disk, initializing [ 1247.175159] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1247.253962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1247.268571] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1252.277324] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1262.253588] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1268.525867] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1275.456170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1275.733987] Lustre: lustre-OST0000: new disk, initializing [ 1275.737495] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1275.744721] Lustre: 47957:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1275.835086] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1282.394570] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1285.192208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1285.202621] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1285.327454] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1291.709066] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1296.725165] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1296.988503] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1302.101396] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1302.311441] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1306.080264] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1306.080488] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1306.088912] LustreError: Skipped 1 previous similar message [ 1306.103547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1306.113476] Lustre: Skipped 1 previous similar message [ 1311.208881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1311.215589] Lustre: Skipped 1 previous similar message [ 1316.325542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1316.329470] Lustre: Skipped 1 previous similar message [ 1318.368163] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1318.562651] Lustre: server umount lustre-OST0000 complete [ 1324.000463] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1324.008740] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1324.034575] Lustre: Skipped 1 previous similar message [ 1324.047605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1326.567849] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1328.576149] Lustre: server umount lustre-MDT0000 complete [ 1332.575035] LustreError: 46791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788165761 with bad export cookie 859457018185351175 [ 1332.586313] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1332.594979] LustreError: 46791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1333.036787] Lustre: server umount lustre-MDT0001 complete [ 1343.810768] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 04:42:50 (1788165770) [ 1350.853932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1351.935119] LustreError: 50419:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1351.966461] LustreError: 50419:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1352.120824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1356.048405] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1357.281341] LustreError: 50420:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1357.309502] LustreError: 50420:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1361.380881] LustreError: 50419:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1364.126714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1364.468104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1369.169812] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1378.091842] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1384.530193] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1391.761284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1392.220699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1398.302767] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1405.035920] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1412.628944] LustreError: 52947:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1413.719830] Lustre: Modifying parameter general.timeout=1 in log params [ 1414.795931] Lustre: Modifying parameter general.timeout=2 in log params [ 1417.776287] Lustre: Modifying parameter general.timeout=5 in log params [ 1417.783902] Lustre: Skipped 2 previous similar messages [ 1422.402413] Lustre: Modifying parameter general.timeout=10 in log params [ 1422.410424] Lustre: Skipped 4 previous similar messages [ 1430.994763] Lustre: Modifying parameter general.timeout=20 in log params [ 1430.997447] Lustre: Skipped 9 previous similar messages [ 1447.218602] Lustre: Modifying parameter general.timeout=38 in log params [ 1447.226395] Lustre: Skipped 17 previous similar messages [ 1480.111420] Lustre: Modifying parameter general.timeout=73 in log params [ 1480.120916] Lustre: Skipped 34 previous similar messages [ 1545.655613] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 04:46:12 (1788165972) [ 1546.406930] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_policies=tbf in log params [ 1546.415531] Lustre: Skipped 57 previous similar messages [ 1548.244902] Lustre: 59895:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1550.531823] Lustre: 60006:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1554.421202] Lustre: 60203:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1554.436176] Lustre: 60203:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1558.929834] Lustre: 60413:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1558.951945] Lustre: 60413:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1587.814578] Lustre: 61775:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1587.846909] LustreError: 61775:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1592.563578] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 04:46:59 (1788166019) [ 1605.088108] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1605.094216] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1605.114396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1605.117266] Lustre: Skipped 2 previous similar messages [ 1609.187916] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1611.510397] Lustre: server umount lustre-MDT0000 complete [ 1614.306216] LustreError: 51147:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1614.332183] LustreError: 51147:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1614.830496] LustreError: 50399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166043 with bad export cookie 859457018185352057 [ 1614.852221] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1614.856412] LustreError: 50399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1615.488678] Lustre: server umount lustre-MDT0001 complete [ 1630.449041] Lustre: server umount lustre-OST0000 complete [ 1662.469165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1662.832474] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1663.343694] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1668.746279] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1678.619621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1678.801458] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1678.851448] Lustre: 64313:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1684.918242] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1694.065662] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1700.599365] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1708.212622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1708.437674] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1708.797876] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1708.803506] Lustre: Skipped 1 previous similar message [ 1716.103899] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1724.449922] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1726.572205] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1726.583724] Lustre: Skipped 22 previous similar messages [ 1726.635630] Lustre: 66881:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1736.674858] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1736.680161] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1736.702206] Lustre: Skipped 1 previous similar message [ 1736.706882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1736.713511] Lustre: Skipped 4 previous similar messages [ 1740.325273] Lustre: server umount lustre-OST0000 complete [ 1745.889201] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1745.900705] LustreError: Skipped 1 previous similar message [ 1750.527109] Lustre: server umount lustre-MDT0000 complete [ 1752.039495] LustreError: 65432:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1752.055508] LustreError: 65432:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1754.320980] LustreError: 64305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166183 with bad export cookie 859457018185355249 [ 1754.352736] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1755.009615] Lustre: server umount lustre-MDT0001 complete [ 1768.383305] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 1771.384797] Key type lgssc unregistered [ 1771.688820] LNet: 68093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1771.698360] LNetError: 68093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1771.719733] LNet: Removed LNI 192.168.203.145@tcp [ 1772.497162] Key type .llcrypt unregistered [ 1772.499374] Key type ._llcrypt unregistered [ 1800.190731] Key type ._llcrypt registered [ 1800.194633] Key type .llcrypt registered [ 1800.353529] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 1801.592845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1801.847115] alg: No test for adler32 (adler32-zlib) [ 1803.016150] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 1803.311090] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 1805.095173] Key type lgssc registered [ 1806.557241] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1814.977547] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1815.015318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1816.546689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1821.595686] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1826.784987] LustreError: 69512:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1826.809799] LustreError: 69512:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1831.124351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1831.782110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1836.502543] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1845.250421] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1851.920494] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1859.224042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1859.950481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1866.889966] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1874.227542] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1885.105070] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1896.938524] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1896.940305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1896.965828] Lustre: Skipped 2 previous similar messages [ 1896.997435] Lustre: Skipped 2 previous similar messages [ 1901.532060] Lustre: server umount lustre-MDT0000 complete [ 1902.053664] LustreError: 69513:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1902.081316] LustreError: 69513:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1905.475984] LustreError: 69492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166334 with bad export cookie 17259816761606332907 [ 1905.485885] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1905.495431] LustreError: 69492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1905.931117] Lustre: server umount lustre-MDT0001 complete [ 1920.553060] Lustre: server umount lustre-OST0000 complete [ 1951.059482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1951.550503] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1951.585896] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1951.781897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1957.106270] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1966.821351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1966.972838] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1967.024378] Lustre: 74385:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1967.305361] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1967.734308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1973.883165] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1981.646166] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1987.753365] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1995.582661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1995.842417] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1996.241186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2003.108454] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2011.950514] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2013.900250] Lustre: 76959:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2015.730743] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2023.919402] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2023.927600] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2023.938684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2024.425276] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2026.784877] Lustre: server umount lustre-OST0000 complete [ 2034.147090] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2034.174341] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2034.198807] Lustre: Skipped 1 previous similar message [ 2034.203334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2034.213414] Lustre: Skipped 2 previous similar messages [ 2039.783109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2039.794505] Lustre: Skipped 2 previous similar messages [ 2044.895441] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2045.085664] Lustre: server umount lustre-MDT0000 complete [ 2049.477351] LustreError: 74375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166478 with bad export cookie 17259816761606333747 [ 2049.487199] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2049.499687] LustreError: 74375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2050.060372] Lustre: server umount lustre-MDT0001 complete [ 2064.120202] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 2068.256365] Key type lgssc unregistered [ 2068.812632] LNet: 78171:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2068.829438] LNetError: 78171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2068.867425] LNet: Removed LNI 192.168.203.145@tcp [ 2070.517738] Key type .llcrypt unregistered [ 2070.525474] Key type ._llcrypt unregistered [ 2101.979984] Key type ._llcrypt registered [ 2101.985222] Key type .llcrypt registered [ 2102.081921] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2103.045568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2103.081412] alg: No test for adler32 (adler32-zlib) [ 2104.218642] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 2104.387421] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 2106.043386] Key type lgssc registered [ 2107.133991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2114.894108] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2114.911581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2116.328689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2121.087584] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2126.826942] LustreError: 79591:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2126.840533] LustreError: 79591:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2129.398373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2130.112536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2134.785584] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2142.500965] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2148.252787] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2155.923249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2156.649829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2163.290328] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2170.616806] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2180.847214] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2181.808991] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2183.590675] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2183.595755] Lustre: Skipped 1 previous similar message [ 2185.904254] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2185.910523] Lustre: Skipped 1 previous similar message [ 2189.986796] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2190.000808] Lustre: Skipped 3 previous similar messages [ 2198.981060] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2198.986450] Lustre: Skipped 8 previous similar messages [ 2215.448231] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2215.455657] Lustre: Skipped 17 previous similar messages [ 2248.276091] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2248.282385] Lustre: Skipped 39 previous similar messages [ 2275.378519] Lustre: 87690:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2277.916885] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 04:58:24 (1788166704) [ 2279.488532] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2281.261445] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 04:58:27 (1788166707) [ 2288.096221] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2288.101647] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2288.119825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2288.133899] Lustre: Skipped 1 previous similar message [ 2288.152814] Lustre: Skipped 1 previous similar message [ 2291.441101] Lustre: server umount lustre-OST0000 complete [ 2298.336764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2298.349785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2298.360277] Lustre: Skipped 1 previous similar message [ 2300.957389] Lustre: server umount lustre-MDT0000 complete [ 2303.457816] LustreError: 80321:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2303.486955] LustreError: 80321:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2304.588682] LustreError: 79571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166733 with bad export cookie 17708706644723471103 [ 2304.595599] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2304.604591] LustreError: 79571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2304.988699] Lustre: server umount lustre-MDT0001 complete [ 2315.700985] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 2318.178595] Key type lgssc unregistered [ 2318.457648] LNet: 88941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2318.476497] LNetError: 88941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2318.490454] LNet: Removed LNI 192.168.203.145@tcp [ 2319.202619] Key type .llcrypt unregistered [ 2319.208075] Key type ._llcrypt unregistered [ 2323.145230] Key type ._llcrypt registered [ 2323.146821] Key type .llcrypt registered [ 2323.230803] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2326.966380] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules [ 2328.058303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2328.077145] alg: No test for adler32 (adler32-zlib) [ 2329.156817] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 2329.373854] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 2331.007796] Key type lgssc registered [ 2332.434068] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2333.796516] LustreError: 90144:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2350.980728] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2355.233275] LustreError: 90144:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2359.115652] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2359.144579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2360.644783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2365.279026] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2371.040506] LustreError: 91167:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2371.064403] LustreError: 91167:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2373.486508] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 05:00:00 (1788166800) [ 2376.162600] LustreError: 91168:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2378.012257] LustreError: 91198:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2378.026601] LustreError: 91198:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2378.035937] LustreError: 91198:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 2378.371119] Lustre: server umount lustre-MDT0000 complete [ 2391.740482] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 2394.691403] Key type lgssc unregistered [ 2395.019964] LNet: 92693:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2395.028740] LNetError: 92693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2395.040907] LNet: Removed LNI 192.168.203.145@tcp [ 2395.909846] Key type .llcrypt unregistered [ 2395.915099] Key type ._llcrypt unregistered [ 2413.557717] Key type ._llcrypt registered [ 2413.561063] Key type .llcrypt registered [ 2413.669485] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2414.887361] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2414.918293] alg: No test for adler32 (adler32-zlib) [ 2415.959661] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 2416.204966] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 2417.887152] Key type lgssc registered [ 2418.877622] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2425.691473] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2425.716755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2427.280841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2432.548694] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2437.607241] LustreError: 93910:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2437.631157] LustreError: 93910:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2442.708349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2442.723115] LustreError: 93911:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2443.234193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2448.905833] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2458.258274] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2464.060422] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2471.201038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2471.766224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2477.522462] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2484.785098] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2539.749868] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 05:02:46 (1788166966) [ 2541.339657] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2543.051509] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 05:02:49 (1788166969) [ 2548.707386] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2548.712251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2548.720109] Lustre: Skipped 2 previous similar messages [ 2548.729614] Lustre: Skipped 1 previous similar message [ 2553.829752] LustreError: 94875:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2553.856032] LustreError: 94875:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2553.924171] Lustre: server umount lustre-MDT0000 complete [ 2556.986508] LustreError: 93892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166985 with bad export cookie 12284408915446709918 [ 2556.988506] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2557.000592] LustreError: 93892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2557.261710] Lustre: server umount lustre-MDT0001 complete [ 2570.780636] Lustre: server umount lustre-OST0000 complete [ 2580.252824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2580.765167] LustreError: 98218:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2580.881803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2584.697687] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2586.082255] LustreError: 98219:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2592.212979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2592.818490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2596.629407] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2603.218666] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2607.953351] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2618.148562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2618.308771] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2618.313976] LustreError: 98207:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2618.330883] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2618.337127] LustreError: 100299:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2618.340488] LustreError: 100299:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2618.343093] LustreError: 100299:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2618.387821] Lustre: server umount lustre-OST0000 complete [ 2618.390446] LustreError: 100299:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2624.327787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2624.536594] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2624.545353] LustreError: 98205:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2624.561816] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2624.574679] LustreError: 100762:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2624.583556] LustreError: 100762:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2624.592965] LustreError: 100762:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2624.633499] Lustre: server umount lustre-OST0000 complete [ 2624.636070] LustreError: 100762:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2633.272965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2633.468897] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2633.473899] Lustre: Found index 0 for lustre-OST0000, updating log [ 2633.477944] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2633.535291] Lustre: lustre-OST0000: new disk, initializing [ 2633.547084] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2633.555966] Lustre: 101296:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2633.802442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2640.709984] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2644.337819] Lustre: Failing over lustre-OST0000 [ 2644.558350] Lustre: server umount lustre-OST0000 complete [ 2648.414733] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2648.418580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2648.441350] Lustre: Skipped 1 previous similar message [ 2653.666983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2653.677389] Lustre: Skipped 2 previous similar messages [ 2654.702553] Lustre: server umount lustre-MDT0000 complete [ 2657.714171] LustreError: 98200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167086 with bad export cookie 12284408915446712662 [ 2657.715392] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2657.725743] LustreError: 98200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2668.170158] Lustre: server umount lustre-MDT0001 complete [ 2675.358618] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 05:05:02 (1788167102) [ 2689.749840] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2698.074935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2698.319640] LustreError: 103643:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2698.335729] LustreError: 103643:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2698.430530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2701.661808] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2709.016636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2713.391827] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2716.160907] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2722.510538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2723.038381] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2724.083895] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2724.094095] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2724.127093] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -16 [ 2729.458845] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:55 [ 2729.472158] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -16 [ 2729.850599] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2734.573466] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:49 [ 2734.584562] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -16 [ 2735.613156] Lustre: lustre-OST0000: Recovery over after 0:11, of 1 clients 1 recovered and 0 were evicted. [ 2735.615841] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2737.237352] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2737.315871] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2737.403424] Lustre: lustre-OST0001: new disk, initializing [ 2737.405989] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2737.411911] Lustre: 105963:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2737.644861] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2737.650583] Lustre: Skipped 2 previous similar messages [ 2742.878990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2742.887287] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2742.892385] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2742.995744] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2749.920129] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2755.551600] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2760.759230] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.45@tcp (stopping) [ 2761.189897] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2761.200278] LustreError: Skipped 1 previous similar message [ 2761.210197] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2763.758641] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2763.768760] Lustre: Skipped 1 previous similar message [ 2765.546748] Lustre: server umount lustre-MDT0001 complete [ 2765.875563] LustreError: 103638:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2765.889436] LustreError: 103638:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2774.096488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2774.220427] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2774.224673] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2774.260514] Lustre: 103632:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2775.363648] LustreError: 107500:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2775.370410] LustreError: 107500:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2775.376595] LustreError: 103635:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2775.472359] LustreError: 107509:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.145@tcp: cfg command failed: rc = -17 [ 2775.483339] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2775.483339] [ 2778.502490] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2780.137238] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2787.886901] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 05:06:54 (1788167214) [ 2804.501935] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2820.970085] Lustre: 109576:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2841.559472] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2844.680804] Lustre: 110711:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2933.215939] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2933.233997] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2933.266454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2933.273618] Lustre: Skipped 4 previous similar messages [ 2936.852500] Lustre: server umount lustre-MDT0000 complete [ 2938.850213] LustreError: 103644:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2938.873243] LustreError: 103644:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 2940.214044] LustreError: 103672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167369 with bad export cookie 12284408915446713593 [ 2940.215512] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2940.229528] LustreError: 103672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2940.427327] Lustre: server umount lustre-MDT0001 complete [ 2953.274621] Lustre: server umount lustre-OST0000 complete [ 2966.510891] Lustre: server umount lustre-OST0001 complete [ 2972.564875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2983.973655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2991.780287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3001.035813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3021.464628] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 3031.868198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3031.885369] Lustre: lustre-MDT0000: reset Object Index mappings [ 3032.373763] LustreError: 114662:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3032.476579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3032.483135] Lustre: Skipped 1 previous similar message [ 3036.905976] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3046.212412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3046.234331] Lustre: lustre-MDT0001: reset Object Index mappings [ 3051.932237] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3055.074805] Lustre: 115811:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3062.603520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3071.163442] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:231 to 0x280000400:257) [ 3071.208691] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3077.674124] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:350 to 0x280000401:386) [ 3080.871471] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3086.878395] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3086.882866] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3087.832835] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3094.793941] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3097.760122] Lustre: 117688:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3147.415126] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 05:12:54 (1788167574) [ 3173.863381] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3173.868199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3173.882648] Lustre: Skipped 5 previous similar messages [ 3173.885994] Lustre: Skipped 4 previous similar messages [ 3179.403579] Lustre: server umount lustre-MDT0000 complete [ 3182.940837] LustreError: 114643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167611 with bad export cookie 12284408915446760486 [ 3182.969436] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3184.101058] LustreError: 115868:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3184.126299] LustreError: 115868:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 3189.513148] Lustre: server umount lustre-MDT0001 complete [ 3199.772840] Lustre: server umount lustre-OST0000 complete [ 3209.613655] Lustre: server umount lustre-OST0001 complete [ 3215.799630] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 3227.443783] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 3263.270197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3263.536776] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3263.570229] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3263.658368] Lustre: lustre-MDT0000: new disk, initializing [ 3263.722832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3263.731594] Lustre: Skipped 3 previous similar messages [ 3263.744941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3267.540749] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3273.531417] Lustre: server umount lustre-MDT0000 complete [ 3284.437567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3284.828220] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3284.872808] Lustre: 121961:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3284.895551] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3289.038114] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3294.184841] Lustre: server umount lustre-MDT0000 complete [ 3303.545446] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 05:15:30 (1788167730) [ 3305.417874] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3307.228811] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 05:15:33 (1788167733) [ 3325.991197] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 3336.549496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3340.888348] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3351.266880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3351.465492] Lustre: 124011:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3351.558879] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3351.564615] Lustre: Skipped 1 previous similar message [ 3351.660050] Lustre: lustre-MDT0001: new disk, initializing [ 3351.868456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3351.889340] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3357.407476] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3362.890340] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3370.261110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3370.521463] Lustre: lustre-OST0000: new disk, initializing [ 3370.525136] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3370.531775] Lustre: 125589:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3372.485269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3372.514928] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3372.689567] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3379.053656] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3389.567103] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3389.758645] Lustre: lustre-OST0001: new disk, initializing [ 3389.761590] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3389.766137] Lustre: 126461:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3391.892727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3391.906030] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3392.014770] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3396.162546] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3405.356699] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3408.694235] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3804.093871] Lustre: 132879:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3804.100817] Lustre: 132879:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3804.104876] Lustre: 132879:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3804.108823] Lustre: 132879:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3804.112759] Lustre: 132879:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3804.123173] Lustre: 132879:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4077.216847] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 05:28:23 (1788168503) [ 4093.412248] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4093.426731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4093.429691] Lustre: Skipped 6 previous similar messages [ 4093.449341] Lustre: Skipped 11 previous similar messages [ 4098.488511] Lustre: server umount lustre-MDT0000 complete [ 4098.534740] LustreError: 125582:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4098.554919] LustreError: 125582:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4102.424291] LustreError: 132938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168531 with bad export cookie 12284408915446828358 [ 4102.427107] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4102.432699] LustreError: 132938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 4103.669168] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4103.670766] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4103.697038] Lustre: Skipped 1 previous similar message [ 4103.709202] Lustre: Skipped 1 previous similar message [ 4107.682797] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4107.695944] Lustre: Skipped 1 previous similar message [ 4108.895595] Lustre: server umount lustre-MDT0001 complete [ 4118.570692] Lustre: server umount lustre-OST0000 complete [ 4128.198634] Lustre: server umount lustre-OST0001 complete [ 4135.612514] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 4148.465848] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 4189.133586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4189.382424] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4189.402785] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4189.484717] Lustre: lustre-MDT0000: new disk, initializing [ 4189.624360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4189.627799] Lustre: Skipped 5 previous similar messages [ 4189.653945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4194.421388] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4204.469919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4204.602788] Lustre: 135930:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4204.669578] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4204.676987] Lustre: Skipped 1 previous similar message [ 4204.768604] Lustre: lustre-MDT0001: new disk, initializing [ 4204.999508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4205.017450] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4209.348434] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4217.767165] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4224.270555] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4232.067153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4232.415852] Lustre: lustre-OST0000: new disk, initializing [ 4232.421435] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4232.428974] Lustre: 137895:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4232.544891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4232.552264] Lustre: Skipped 1 previous similar message [ 4234.002434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4234.016524] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4234.108285] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4239.097739] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4249.603446] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4259.912495] Lustre: lustre-MDD0000: changelog on [ 4262.495748] Lustre: lustre-MDD0001: changelog on [ 6404.351967] Lustre: lustre-MDD0001: changelog off [ 6406.787766] Lustre: 135935:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6407.739675] Lustre: lustre-MDD0000: changelog off [ 6415.335206] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6415.345558] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6415.358937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6415.372052] Lustre: Skipped 1 previous similar message [ 6420.450478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6420.459923] Lustre: Skipped 1 previous similar message [ 6425.572550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6425.582974] Lustre: Skipped 1 previous similar message [ 6429.151148] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6429.582245] Lustre: server umount lustre-OST0000 complete [ 6435.809693] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6435.816390] Lustre: Skipped 1 previous similar message [ 6435.820205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6435.824688] Lustre: Skipped 1 previous similar message [ 6440.664953] Lustre: server umount lustre-MDT0000 complete [ 6440.947077] LustreError: 135940:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6440.972084] LustreError: 135940:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6444.249761] LustreError: 135920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788170873 with bad export cookie 12284408915447607584 [ 6444.259294] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6444.263653] LustreError: 135920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6444.653847] Lustre: server umount lustre-MDT0001 complete [ 6456.493478] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 6459.221773] Key type lgssc unregistered [ 6459.654706] LNet: 147086:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6459.673910] LNetError: 147086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6459.711243] LNet: Removed LNI 192.168.203.145@tcp [ 6460.894253] Key type .llcrypt unregistered [ 6460.897846] Key type ._llcrypt unregistered [ 6483.758188] Key type ._llcrypt registered [ 6483.762235] Key type .llcrypt registered [ 6483.911168] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 6500.249876] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 6501.863963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6501.924389] alg: No test for adler32 (adler32-zlib) [ 6503.059735] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 6503.288930] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 6504.943173] Key type lgssc registered [ 6506.167178] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6545.576058] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6545.629369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6546.881565] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6546.921188] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6546.983492] Lustre: lustre-MDT0000: new disk, initializing [ 6547.041941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6547.056822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6552.100231] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6563.585704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6563.786309] Lustre: 150288:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6563.905506] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6563.913018] Lustre: Skipped 1 previous similar message [ 6564.015655] Lustre: lustre-MDT0001: new disk, initializing [ 6564.160988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6564.209692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6564.225099] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6569.499579] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6579.766113] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6585.975093] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6593.153869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6593.525860] Lustre: lustre-OST0000: new disk, initializing [ 6593.539844] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6593.555445] Lustre: 152257:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6593.672280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6599.274360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6599.298808] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6599.406074] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6600.991535] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6611.661750] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6619.172600] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_module obdecho/obdecho [ 6621.174785] LustreError: 153261:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6621.199205] LustreError: 153261:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6623.100922] LustreError: 153358:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6640.617588] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6640.637079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6640.648410] Lustre: Skipped 1 previous similar message [ 6642.847239] Lustre: server umount lustre-MDT0000 complete [ 6647.443839] LustreError: 152256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788171076 with bad export cookie 7649115530991727129 [ 6647.463723] LustreError: 152256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6647.468982] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6648.599712] Lustre: server umount lustre-MDT0001 complete [ 6664.759406] Lustre: server umount lustre-OST0000 complete [ 6690.469375] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 6700.791099] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 6741.593090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6741.807527] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6741.832662] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6741.893792] Lustre: lustre-MDT0000: new disk, initializing [ 6741.974366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6742.011252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6746.360065] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6757.231214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6757.333838] Lustre: 157077:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6757.384596] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6757.387593] Lustre: Skipped 1 previous similar message [ 6757.439126] Lustre: lustre-MDT0001: new disk, initializing [ 6757.491297] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6757.515753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6757.527549] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6761.497077] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6771.134238] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6777.104487] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6783.747335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6783.957940] Lustre: lustre-OST0000: new disk, initializing [ 6783.962126] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6783.968893] Lustre: 159045:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6784.085466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6791.135346] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6792.238719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6792.247190] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6792.365240] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6800.846772] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6805.583946] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6805.857750] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6810.787983] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6810.969814] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6813.164522] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6813.166229] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6813.182728] LustreError: Skipped 1 previous similar message [ 6813.185914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6813.242238] Lustre: Skipped 3 previous similar messages [ 6818.277836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6818.282060] Lustre: Skipped 1 previous similar message [ 6819.250995] Lustre: server umount lustre-OST0000 complete [ 6824.435735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6824.444386] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6824.463908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6824.469647] Lustre: Skipped 1 previous similar message [ 6828.514832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6828.517247] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6830.680522] Lustre: server umount lustre-MDT0000 complete [ 6833.635625] LustreError: 157889:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6833.661562] LustreError: 157889:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6834.664507] LustreError: 157067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788171263 with bad export cookie 7649115530991728088 [ 6834.669720] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6834.683467] LustreError: 157067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6834.959601] Lustre: server umount lustre-MDT0001 complete [ 6839.209611] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 06:14:25 (1788171265) [ 6840.787803] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6842.618908] Lustre: DEBUG MARKER: == conf-sanity test 138: MDT mount failure must not LBUG destroying recovery exports ========================================================== 06:14:29 (1788171269) [ 6864.161598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6864.475709] LustreError: 162016:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6864.539654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6869.296381] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6869.989301] LustreError: 162017:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6875.106333] LustreError: 162016:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6878.383661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6883.378564] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6891.077932] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6898.080561] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6905.859828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6906.132071] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6906.141518] Lustre: Skipped 1 previous similar message [ 6913.284376] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6921.939281] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6930.718854] Lustre: *** cfs_fail_loc=718, val=0*** [ 6930.983745] Lustre: server umount lustre-MDT0000 complete [ 6933.080163] LustreError: 163257:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6933.990712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6934.001553] Lustre: Skipped 2 previous similar messages [ 6934.952049] LustreError: 161996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788171363 with bad export cookie 7649115530991728970 [ 6934.953289] Lustre: *** cfs_fail_loc=718, val=0*** [ 6934.977673] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6934.980376] Lustre: Skipped 3 previous similar messages [ 6935.503281] Lustre: server umount lustre-MDT0001 complete [ 6946.554807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6954.463600] Lustre: 148512:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788171367/real 1788171367] req@000000006e680c21 x1875033123308672/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788171383 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6954.506044] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6959.719984] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x6a271e0c76af7c6f [ 6959.739737] Lustre: MGC192.168.203.145@tcp: Connection restored to 0@lo (at 0@lo) [ 6960.017594] Lustre: *** cfs_fail_loc=723, val=0*** [ 6960.023875] Lustre: Skipped 1 previous similar message [ 6960.156629] LustreError: 165374:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-122) [ 6960.167554] LustreError: 165374:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.145@tcp: cfg command failed: rc = -122 [ 6960.184746] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 6960.184746] [ 6960.197803] LustreError: MGC192.168.203.145@tcp: Confguration from log lustre-MDT0000 failed from MGS -122. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 6960.216596] LustreError: 165343:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -122 [ 6960.227853] LustreError: 165343:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -122 [ 6960.240850] LustreError: 165343:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 6960.253473] LustreError: 165343:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 6960.403916] Lustre: server umount lustre-MDT0000 complete [ 6960.412196] LustreError: 165343:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -122 [ 6961.249218] Lustre: 148511:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788171373/real 1788171373] req@00000000981ef02d x1875033123308800/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788171389 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6965.407641] Lustre: 148511:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788171378/real 1788171378] req@00000000305954d2 x1875033123309056/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788171394 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6969.248371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6969.385365] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6969.406412] LustreError: 162008:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 6969.850600] LustreError: 163848:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6969.887530] LustreError: 163848:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6970.030714] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6970.463747] Lustre: 148512:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788171382/real 1788171382] req@0000000090f6e6fa x1875033123309312/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788171398 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6973.032165] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6974.061045] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6975.011579] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6985.473166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6985.770763] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 6985.896830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6985.907266] Lustre: Skipped 1 previous similar message [ 6985.942422] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 6986.383483] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6991.347380] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6991.554571] Lustre: lustre-MDT0000: Recovery over after 0:18, of 2 clients 2 recovered and 0 were evicted. [ 6992.890256] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7002.334619] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7004.358915] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7010.795608] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7012.229070] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7016.665611] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 7032.307469] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7032.312052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7032.321515] Lustre: Skipped 1 previous similar message [ 7032.336701] Lustre: Skipped 2 previous similar messages [ 7036.320989] Lustre: server umount lustre-OST0000 complete [ 7052.773326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7052.799780] Lustre: Skipped 5 previous similar messages [ 7055.327123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7055.510459] Lustre: server umount lustre-MDT0000 complete [ 7057.891925] LustreError: 165780:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7057.903836] LustreError: 165780:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 7059.646983] LustreError: 161997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788171488 with bad export cookie 7649115530991729922 [ 7059.647785] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7059.663448] LustreError: 161997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 7059.991152] Lustre: server umount lustre-MDT0001 complete [ 7071.407904] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 7074.364904] Key type lgssc unregistered [ 7074.712316] LNet: 168855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7074.718369] LNetError: 168855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7074.737214] LNet: Removed LNI 192.168.203.145@tcp [ 7075.580392] Key type .llcrypt unregistered [ 7075.589555] Key type ._llcrypt unregistered [ 7106.131935] Key type ._llcrypt registered [ 7106.136867] Key type .llcrypt registered [ 7106.251984] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 7107.387210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7107.481542] alg: No test for adler32 (adler32-zlib) [ 7108.551758] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 7108.895843] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 7110.575381] Key type lgssc registered [ 7111.626795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7119.257643] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7119.297908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7120.858689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7125.276823] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7131.105916] LustreError: 170725:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7131.127799] LustreError: 170725:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7134.385574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7134.850931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7139.791391] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7150.086367] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7156.401105] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7164.745443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7165.267719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7166.256465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7172.486626] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7181.508929] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7187.922312] Lustre: *** cfs_fail_loc=718, val=0*** [ 7188.283101] Lustre: server umount lustre-MDT0000 complete [ 7188.557204] LustreError: 170720:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7189.986043] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7189.999540] LustreError: 170722:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7190.011226] Lustre: Skipped 1 previous similar message [ 7190.036039] LustreError: 170722:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7192.219909] Lustre: *** cfs_fail_loc=718, val=0*** [ 7192.223375] LustreError: 170705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788171621 with bad export cookie 14132457281964553696 [ 7192.230439] Lustre: Skipped 3 previous similar messages [ 7192.234641] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7192.248842] LustreError: 170705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7192.565553] Lustre: server umount lustre-MDT0001 complete [ 7205.778192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7210.399149] Lustre: 170039:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788171623/real 1788171623] req@0000000010bec786 x1875033757611776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788171639 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7210.424988] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7210.438702] Lustre: Skipped 1 previous similar message [ 7213.535356] Lustre: 170039:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788171626/real 1788171626] req@00000000ef442e44 x1875033757611904/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788171642 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7216.607520] Lustre: 170037:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788171629/real 1788171629] req@000000006682d44c x1875033757612032/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788171645 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7217.638552] LustreError: 170035:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c28ebed5 x1875033757612928/t0(0) o250->MGC192.168.203.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7218.113476] Lustre: *** cfs_fail_loc=727, val=0*** [ 7218.232453] LustreError: 174129:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-57) [ 7218.240951] LustreError: 174129:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.145@tcp: cfg command failed: rc = -57 [ 7218.253620] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 7218.253620] [ 7218.286091] LustreError: MGC192.168.203.145@tcp: Confguration from log lustre-MDT0000 failed from MGS -57. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 7218.307057] LustreError: 174099:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -57 [ 7218.316092] LustreError: 174099:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -57 [ 7218.336347] LustreError: 174099:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 7218.343200] LustreError: 174099:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 7218.537826] Lustre: server umount lustre-MDT0000 complete [ 7218.549560] LustreError: 174099:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -57 [ 7218.687145] Lustre: 170036:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788171631/real 1788171631] req@00000000dd56e0c9 x1875033757612288/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788171647 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7223.711619] Lustre: 170039:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788171636/real 1788171636] req@00000000948012f1 x1875033757612416/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788171652 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7229.695785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7229.834482] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7229.848488] LustreError: 170717:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 7230.147978] LustreError: 172555:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7230.249488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7230.317100] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7231.283526] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7234.661062] LustreError: 174632:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7234.684437] LustreError: 174632:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7236.085827] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7244.836891] LustreError: 174632:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7244.864656] LustreError: 174632:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7246.453182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7246.789444] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 7246.910515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7246.954609] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 7249.954815] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7251.912061] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7251.948963] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7251.951713] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7252.124814] Lustre: lustre-MDT0001: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 7253.072842] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 7253.073843] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7253.091916] Lustre: Skipped 1 previous similar message [ 7253.150606] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7259.873299] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7261.310361] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7267.301210] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7268.832650] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7273.198183] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 7284.193603] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7284.206827] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7284.219923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7287.788117] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7287.788851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7287.815879] Lustre: Skipped 1 previous similar message [ 7289.715756] Lustre: server umount lustre-OST0000 complete [ 7294.435330] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7294.440873] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7294.454099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7298.017431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7298.033796] Lustre: Skipped 1 previous similar message [ 7299.596510] Lustre: server umount lustre-MDT0000 complete [ 7303.139577] LustreError: 174868:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7303.153944] LustreError: 174868:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7303.383375] LustreError: 171461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788171732 with bad export cookie 14132457281964554893 [ 7303.392830] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7303.395875] LustreError: 171461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7303.916548] Lustre: server umount lustre-MDT0001 complete [ 7315.065793] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 7318.112845] Key type lgssc unregistered [ 7318.467050] LNet: 177706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7318.479234] LNetError: 177706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7318.499958] LNet: Removed LNI 192.168.203.145@tcp [ 7319.552367] Key type .llcrypt unregistered [ 7319.558614] Key type ._llcrypt unregistered [ 7339.411222] Key type ._llcrypt registered [ 7339.415116] Key type .llcrypt registered [ 7339.505868] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 7340.473863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7340.504116] alg: No test for adler32 (adler32-zlib) [ 7341.537272] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 7341.794775] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 7343.479240] Key type lgssc registered [ 7344.604623] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7352.554148] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7352.582315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7354.140401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7359.022445] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7363.555062] LustreError: 179027:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7363.588586] LustreError: 179027:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7368.129849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7368.652865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7372.970959] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7380.402466] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7385.813965] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7392.735548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7393.129344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7398.268468] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7400.136256] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7407.977882] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7418.853980] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7418.875124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7418.888756] Lustre: Skipped 1 previous similar message [ 7421.018920] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.45@tcp (stopping) [ 7421.029526] Lustre: Skipped 2 previous similar messages [ 7421.363675] Lustre: server umount lustre-MDT0000 complete [ 7423.972856] LustreError: 179023:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7423.993224] LustreError: 179023:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7425.436936] LustreError: 179008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788171854 with bad export cookie 7876969613577741084 [ 7425.446839] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7425.449707] LustreError: 179008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7426.088560] LustreError: 180460:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7426.166277] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.45@tcp (stopping) [ 7429.096061] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7429.102547] LustreError: 180460:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7429.108741] Lustre: Skipped 1 previous similar message [ 7429.162202] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7431.993575] Lustre: server umount lustre-MDT0001 complete [ 7439.739588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7454.066752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7454.293362] LustreError: 182871:0:(llog.c:1655:llog_backup()) MGC192.168.203.145@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7454.296943] Lustre: 182871:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.145@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7455.715507] LustreError: 182876:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7455.740554] LustreError: 182876:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7465.952347] LustreError: 182877:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7465.978854] LustreError: 182877:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 7482.926172] LustreError: 182877:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7482.967966] LustreError: 182877:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 7515.104833] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7515.115407] Lustre: 182871:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.145@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7516.641288] LustreError: 180858:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7516.652859] LustreError: 180858:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 23 previous similar messages [ 7556.086061] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7560.371362] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7561.195437] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7568.033546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7570.207124] LustreError: 183616:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 7570.227085] LustreError: 183616:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@000000008f1936c5 x1875034001940736/t0(0) o250->MGC192.168.203.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1788171999 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7570.247384] LustreError: 183616:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 7570.596153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7570.664056] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:193) [ 7574.474148] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7576.062723] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7581.573728] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7582.987979] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7588.193819] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7589.584652] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7611.084500] Lustre: server umount lustre-MDT0000 complete [ 7611.874387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7611.876028] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7611.898039] Lustre: Skipped 2 previous similar messages [ 7611.899919] LustreError: 182876:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7611.929617] LustreError: 182876:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 41 previous similar messages [ 7614.622770] LustreError: 179009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788172043 with bad export cookie 7876969613577747755 [ 7614.623052] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7614.639981] LustreError: 179009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7615.003679] Lustre: server umount lustre-MDT0001 complete [ 7629.699432] Lustre: server umount lustre-OST0000 complete [ 7650.561362] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 7659.959483] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 7698.082115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7698.360155] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7698.402689] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7698.496648] Lustre: lustre-MDT0000: new disk, initializing [ 7698.586785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7698.611175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7703.817151] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7713.621219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7713.807518] Lustre: 187977:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7713.885319] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7713.890043] Lustre: Skipped 1 previous similar message [ 7713.971408] Lustre: lustre-MDT0001: new disk, initializing [ 7714.138457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7714.148601] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7719.002709] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7728.861450] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7734.731990] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7741.882571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7742.181605] Lustre: lustre-OST0000: new disk, initializing [ 7742.188788] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7742.197161] Lustre: 189943:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7742.315583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7742.329378] Lustre: Skipped 1 previous similar message [ 7747.675255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7747.686398] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7747.791338] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7748.946035] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7759.362391] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7764.236268] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7764.437367] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7768.906268] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7769.169766] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7773.666474] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7773.672051] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7773.686015] LustreError: Skipped 1 previous similar message [ 7773.688372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7773.696228] Lustre: Skipped 3 previous similar messages [ 7777.411710] Lustre: server umount lustre-OST0000 complete [ 7783.905572] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7783.940497] Lustre: Skipped 1 previous similar message [ 7783.962541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7783.978787] Lustre: Skipped 1 previous similar message [ 7788.327778] Lustre: server umount lustre-MDT0000 complete [ 7789.035820] LustreError: 187989:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7789.063836] LustreError: 187989:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7792.497632] LustreError: 187969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788172221 with bad export cookie 7876969613577764814 [ 7792.503554] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7792.515044] LustreError: 187969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7793.008951] Lustre: server umount lustre-MDT0001 complete [ 7803.436513] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 06:30:29 (1788172229) [ 7811.209745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7811.774727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7816.437561] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7825.790857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7831.270333] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7838.634408] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7844.429331] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7851.408156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7858.936894] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7866.774147] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7876.781474] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 06:31:43 (1788172303) [ 7882.727910] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7882.743494] Lustre: Skipped 1 previous similar message [ 7882.767939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7887.164196] Lustre: server umount lustre-OST0000 complete [ 7892.964665] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7896.925744] Lustre: server umount lustre-MDT0000 complete [ 7900.803446] LustreError: 192339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788172329 with bad export cookie 7876969613577765689 [ 7900.810313] LustreError: 192339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7900.813070] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7901.142126] Lustre: server umount lustre-MDT0001 complete [ 7912.030252] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 7914.682259] Key type lgssc unregistered [ 7914.948621] LNet: 196087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7914.973082] LNetError: 196087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7915.004043] LNet: Removed LNI 192.168.203.145@tcp [ 7915.899463] Key type .llcrypt unregistered [ 7915.901311] Key type ._llcrypt unregistered [ 7936.604332] Key type ._llcrypt registered [ 7936.606223] Key type .llcrypt registered [ 7936.726307] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 7937.865220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7937.922754] alg: No test for adler32 (adler32-zlib) [ 7939.021680] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 7939.239867] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 7940.965438] Key type lgssc registered [ 7942.027261] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7950.277335] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7950.295517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7965.858589] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7970.975314] LustreError: 197335:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.145@tcp: failed processing log, type 4: rc = -110 [ 7976.095550] LustreError: 197335:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7976.125164] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7976.137795] LustreError: MGC192.168.203.145@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7976.160149] LustreError: 197335:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7976.184852] LustreError: 197335:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7976.191604] LustreError: 197335:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7976.265926] Lustre: server umount lustre-OST0000 complete [ 7976.273448] LustreError: 197335:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7986.015201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7987.423361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7993.035858] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7997.930023] LustreError: 197835:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7997.949903] LustreError: 197835:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8002.691944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8003.042294] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8003.288709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8008.189318] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8016.046549] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8022.078052] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8029.167838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8029.475746] LustreError: 199652:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8029.487957] Lustre: 199652:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.145@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8029.639504] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8036.249900] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8044.522311] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8048.098773] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8048.107538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8048.123610] Lustre: Skipped 1 previous similar message [ 8048.142273] Lustre: Skipped 2 previous similar messages [ 8049.201201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8049.209654] Lustre: Skipped 1 previous similar message [ 8053.073146] Lustre: server umount lustre-MDT0000 complete [ 8054.242956] LustreError: 197829:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8054.285513] LustreError: 197829:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8057.218628] LustreError: 197816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788172486 with bad export cookie 117543159713776252 [ 8057.240187] LustreError: 197816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8057.241581] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8057.412684] LustreError: 198565:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000086675bf7 x1875034628976640/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8057.632741] Lustre: server umount lustre-MDT0001 complete [ 8071.726646] Lustre: server umount lustre-OST0000 complete [ 8081.787949] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 8088.855045] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 8102.971335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8103.080239] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8103.096192] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8103.135565] Lustre: lustre-MDT0000: new disk, initializing [ 8103.167433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8103.178599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8104.954285] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8109.385905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8109.431302] Lustre: 203172:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8109.455056] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8109.456920] Lustre: Skipped 1 previous similar message [ 8109.492542] Lustre: lustre-MDT0001: new disk, initializing [ 8109.526569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8109.540502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8109.544660] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8110.977239] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8114.722236] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8116.563172] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8118.718177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8118.807551] Lustre: lustre-OST0000: new disk, initializing [ 8118.809411] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8118.811575] Lustre: 205141:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8118.838578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8120.317981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8120.320952] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8120.355982] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8120.608263] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8124.451139] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8125.826243] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8125.899762] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8127.239061] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8127.312249] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8130.066779] Lustre: server umount lustre-OST0000 complete [ 8135.135721] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8135.139345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8135.145687] Lustre: Skipped 1 previous similar message [ 8135.148348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8140.768370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8140.770977] Lustre: Skipped 2 previous similar messages [ 8141.825566] Lustre: server umount lustre-MDT0000 complete [ 8143.084989] LustreError: 203164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788172571 with bad export cookie 117543159713776973 [ 8143.087619] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8143.089048] LustreError: 203164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8152.583601] Lustre: server umount lustre-MDT0001 complete [ 8155.091264] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 06:36:22 (1788172582) [ 8160.571776] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 8161.602329] Key type lgssc unregistered [ 8161.723409] LNet: 208083:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8161.725532] LNetError: 208083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8161.737264] LNet: Removed LNI 192.168.203.145@tcp [ 8162.035727] Key type .llcrypt unregistered [ 8162.037206] Key type ._llcrypt unregistered [ 8168.736742] Key type ._llcrypt registered [ 8168.737798] Key type .llcrypt registered [ 8168.767595] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 8169.082632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8169.138237] alg: No test for adler32 (adler32-zlib) [ 8169.976367] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 8170.055637] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 8171.631126] Key type lgssc registered [ 8171.963131] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8174.315280] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8174.320477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8175.457087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8176.803575] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8179.729835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8179.858108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8181.331260] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8183.866350] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8185.855721] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8188.382507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8188.446040] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8188.447269] LustreError: 211138:0:(llog.c:1696:llog_backup()) MGC192.168.203.145@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8188.449655] Lustre: 211138:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.145@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8188.488814] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8190.353815] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8192.744810] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8195.043071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8195.044673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8195.049102] Lustre: Skipped 2 previous similar messages [ 8195.051552] Lustre: Skipped 1 previous similar message [ 8199.580355] Lustre: server umount lustre-MDT0000 complete [ 8200.159994] LustreError: 210709:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8200.166573] LustreError: 210709:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 8200.744415] LustreError: 209258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788172629 with bad export cookie 10828325866561735698 [ 8200.747226] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8200.750143] LustreError: 209258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8200.853903] Lustre: server umount lustre-MDT0001 complete [ 8211.217131] Lustre: server umount lustre-OST0000 complete [ 8214.381749] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 8218.099701] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 8229.651627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8229.722524] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8229.730673] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8229.761796] Lustre: lustre-MDT0000: new disk, initializing [ 8229.779550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8229.784583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8231.191289] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8235.208902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8235.249985] Lustre: 214656:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8235.267906] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8235.269240] Lustre: Skipped 1 previous similar message [ 8235.270777] LustreError: 215440:0:(llog.c:1696:llog_backup()) MGC192.168.203.145@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8235.274296] LustreError: 215440:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8235.276109] Lustre: 215440:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.145@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8235.283894] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8235.285819] Lustre: Skipped 1 previous similar message [ 8235.320909] Lustre: lustre-MDT0001: new disk, initializing [ 8235.352178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8235.363684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8235.367025] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8236.596791] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8240.254849] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8242.021464] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8244.255334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8244.338902] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8244.340208] Lustre: Skipped 1 previous similar message [ 8244.341763] LustreError: 216601:0:(llog.c:1696:llog_backup()) MGC192.168.203.145@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8244.344784] LustreError: 216601:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8244.346611] Lustre: 216601:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.145@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8244.350327] Lustre: 216601:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8244.358498] Lustre: lustre-OST0000: new disk, initializing [ 8244.359868] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8244.363122] Lustre: 216626:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8244.390365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8246.242629] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8249.788628] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8249.848181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8249.852390] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8249.884599] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8251.091282] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8251.158275] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8252.395115] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8252.461389] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8254.945839] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8254.946036] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8254.947943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8254.949559] LustreError: Skipped 1 previous similar message [ 8254.956419] Lustre: Skipped 1 previous similar message [ 8259.149775] Lustre: server umount lustre-OST0000 complete [ 8261.087871] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8261.090815] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8261.095789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8261.098815] Lustre: Skipped 1 previous similar message [ 8265.184119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8265.184633] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8266.584095] Lustre: server umount lustre-MDT0000 complete [ 8267.821543] LustreError: 216624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788172696 with bad export cookie 10828325866561736398 [ 8267.824944] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8267.825654] LustreError: 216624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8267.938120] Lustre: server umount lustre-MDT0001 complete [ 8270.911969] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 06:38:18 (1788172698) [ 8276.291566] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 8279.488970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8279.599267] LustreError: 219746:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8279.611287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8280.832604] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8283.701444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8285.163848] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8286.070503] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8288.120835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8290.044369] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8292.732132] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8292.788921] Lustre: lustre-OST0001: new disk, initializing [ 8292.790423] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8292.793209] Lustre: 222053:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8294.592570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8294.596171] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8294.606447] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8294.628724] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8299.866712] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8304.393799] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8304.987742] Lustre: DEBUG MARKER: ADD OST3 [ 8306.513913] loop: module loaded [ 8306.529860] loop0: detected capacity change from 0 to 204800000 [ 8306.614331] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8306.618186] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8306.623289] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8308.484519] loop0: detected capacity change from 0 to 204800000 [ 8310.187268] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8310.253897] Lustre: lustre-OST0003: new disk, initializing [ 8310.256473] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8310.259834] Lustre: 223783:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8311.641409] Lustre: DEBUG MARKER: STOP OST3 [ 8311.884682] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8311.914337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8311.917265] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8311.927373] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8312.266845] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8312.288713] Lustre: Failing over lustre-OST0003 [ 8312.326125] Lustre: server umount lustre-OST0003 complete [ 8313.951052] LustreError: 224126:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8316.408827] Lustre: DEBUG MARKER: START OST3 again [ 8317.412801] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8317.414866] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8317.419347] LustreError: 221419:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8317.427132] LustreError: 221419:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8318.570517] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8318.650851] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8318.654136] Lustre: Skipped 4 previous similar messages [ 8318.657893] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8320.100504] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8320.171227] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8320.171264] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8320.483347] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8334.012544] Lustre: Failing over lustre-OST0003 [ 8334.053036] Lustre: server umount lustre-OST0003 complete [ 8334.315684] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8334.317530] LustreError: 225355:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8334.321228] Lustre: Skipped 2 previous similar messages [ 8334.327975] LustreError: 225355:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8339.429857] LustreError: 225355:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8339.436557] LustreError: 225355:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8349.668235] LustreError: 221230:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8349.671704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8349.674285] LustreError: 221230:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8349.676667] Lustre: Skipped 4 previous similar messages [ 8352.805106] Lustre: server umount lustre-MDT0000 complete [ 8354.105347] LustreError: 220481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788172782 with bad export cookie 10828325866561737266 [ 8354.109059] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8354.110934] LustreError: 220481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8354.784539] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8354.788486] Lustre: Skipped 4 previous similar messages [ 8359.904347] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8359.907502] Lustre: Skipped 3 previous similar messages [ 8360.303873] Lustre: server umount lustre-MDT0001 complete [ 8361.671476] Lustre: server umount lustre-OST0000 complete [ 8362.957300] Lustre: server umount lustre-OST0001 complete [ 8365.170355] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 8368.323599] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 8380.786804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8380.860497] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8380.869461] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8380.903695] Lustre: lustre-MDT0000: new disk, initializing [ 8380.927576] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8382.177481] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8386.162059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8386.202033] Lustre: 229194:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8386.287326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8386.289344] Lustre: Skipped 1 previous similar message [ 8386.300903] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8387.607545] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8391.282463] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8393.078597] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8395.273920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8395.375264] Lustre: 231161:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8396.405768] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8396.442487] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8397.289912] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8400.989819] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8402.352687] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8402.418294] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8403.808794] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8403.883550] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8406.605386] Lustre: server umount lustre-OST0000 complete [ 8412.128993] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8412.129091] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8412.129604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8412.139157] Lustre: Skipped 2 previous similar messages [ 8422.367122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8422.367152] Lustre: 208590:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788172835/real 1788172835] req@ffff890630807100 x1875034870300544/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788172851 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp-pre-0-1.0' uid:0 gid:0 projid:4294967295 [ 8422.433126] Lustre: server umount lustre-MDT0000 complete [ 8423.710885] LustreError: 229188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788172852 with bad export cookie 10828325866561739520 [ 8423.714631] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8423.715538] LustreError: 229188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8423.828885] Lustre: server umount lustre-MDT0001 complete [ 8425.021472] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 06:40:52 (1788172852) [ 8430.185154] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 8433.519202] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 8444.985505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8445.064572] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8445.075381] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8445.077900] Lustre: Skipped 3 previous similar messages [ 8445.110048] Lustre: lustre-MDT0000: new disk, initializing [ 8445.111530] Lustre: Skipped 2 previous similar messages [ 8445.135793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8445.138918] Lustre: Skipped 2 previous similar messages [ 8446.586299] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8450.749948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8450.790786] Lustre: 235764:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8450.892603] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8452.213250] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8455.930272] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8457.731330] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8459.950876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8460.050242] Lustre: 237730:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8461.852596] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8462.013180] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8465.654810] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8467.060066] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8467.133398] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8468.493939] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8468.565973] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8471.312233] Lustre: server umount lustre-OST0000 complete [ 8476.639864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8476.642841] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8476.647973] Lustre: Skipped 1 previous similar message [ 8476.649779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8476.652050] Lustre: Skipped 5 previous similar messages [ 8482.299301] Lustre: server umount lustre-MDT0000 complete [ 8483.527288] LustreError: 235798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788172912 with bad export cookie 10828325866561740402 [ 8483.530888] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8483.532874] LustreError: 235798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8494.095583] Lustre: server umount lustre-MDT0001 complete [ 8497.194537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8497.370934] LustreError: 239944:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8497.378100] LustreError: 239944:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 8498.909111] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8501.937450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8503.503571] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8505.996698] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8507.924358] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8510.259060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8512.214382] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8514.526628] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8531.935872] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8537.438307] Lustre: server umount lustre-OST0000 complete [ 8543.202901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8543.205535] Lustre: Skipped 8 previous similar messages [ 8544.796311] Lustre: server umount lustre-MDT0000 complete [ 8546.024222] LustreError: 239925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788172974 with bad export cookie 10828325866561741284 [ 8546.028196] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8546.028523] LustreError: 239925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8546.145711] Lustre: server umount lustre-MDT0001 complete [ 8648.373966] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 8649.462343] Key type lgssc unregistered [ 8649.595527] LNet: 243628:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8649.599405] LNetError: 243628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8649.606284] LNet: Removed LNI 192.168.203.145@tcp [ 8649.939138] Key type .llcrypt unregistered [ 8649.940609] Key type ._llcrypt unregistered [ 8658.207092] Key type ._llcrypt registered [ 8658.208333] Key type .llcrypt registered [ 8658.242550] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 8664.589767] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 8664.920465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8664.929429] alg: No test for adler32 (adler32-zlib) [ 8665.786443] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 8665.874473] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 8667.455162] Key type lgssc registered [ 8667.821740] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8680.059883] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8680.066793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8681.147348] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8681.158624] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8681.194177] Lustre: lustre-MDT0000: new disk, initializing [ 8681.215887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8681.222629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8682.600803] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8686.959650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8687.006987] Lustre: 246968:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8687.033206] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8687.035919] Lustre: Skipped 1 previous similar message [ 8687.074255] Lustre: lustre-MDT0001: new disk, initializing [ 8687.109616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8687.125050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8687.129776] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8688.553675] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8692.404731] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8694.259085] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8696.684349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8696.786320] Lustre: lustre-OST0000: new disk, initializing [ 8696.787978] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8696.791195] Lustre: 248933:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8696.821354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8698.735299] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8699.638073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8699.642202] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8699.669599] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8702.381344] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8703.780700] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8703.869249] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8705.284386] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8705.368824] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8710.115058] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8710.116041] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8710.118589] LustreError: Skipped 1 previous similar message [ 8710.124277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8710.128506] Lustre: Skipped 1 previous similar message [ 8715.234140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8715.236142] Lustre: Skipped 1 previous similar message [ 8720.351114] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8720.353044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8720.356837] Lustre: Skipped 1 previous similar message [ 8720.417453] Lustre: server umount lustre-OST0000 complete [ 8722.911816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8722.914499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8722.918952] Lustre: Skipped 1 previous similar message [ 8722.921404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8725.472785] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8730.592376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8730.594347] Lustre: Skipped 3 previous similar messages [ 8736.223168] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8736.292634] Lustre: server umount lustre-MDT0000 complete [ 8737.573158] LustreError: 246961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788173166 with bad export cookie 1278875278001124520 [ 8737.575977] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8737.578118] LustreError: 246961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8737.682157] Lustre: server umount lustre-MDT0001 complete [ 8743.620552] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 8746.930072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8747.064768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8748.426865] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8751.247186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8751.363538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8752.778656] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8753.735177] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8755.910968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8757.895343] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8760.541725] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8760.599501] Lustre: lustre-OST0001: new disk, initializing [ 8760.602027] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8760.605606] Lustre: 254170:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8760.633711] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8760.635856] Lustre: Skipped 1 previous similar message [ 8760.984997] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8760.989869] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8761.047968] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8762.511633] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8767.950640] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8770.526096] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8774.170856] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 06:46:41 (1788173201) [ 8777.183931] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8777.187286] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8777.193318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8777.195346] Lustre: Skipped 2 previous similar messages [ 8781.017109] Lustre: server umount lustre-MDT0000 complete [ 8782.178050] LustreError: 251846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788173211 with bad export cookie 1278875278001125388 [ 8782.181601] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8782.182360] LustreError: 251846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8782.293921] Lustre: server umount lustre-MDT0001 complete [ 8793.731322] Lustre: server umount lustre-OST0000 complete [ 8804.246418] Lustre: server umount lustre-OST0001 complete [ 8806.163296] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 8809.269254] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 8820.170617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8820.249156] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8820.258319] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8820.289702] Lustre: lustre-MDT0000: new disk, initializing [ 8820.309466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8820.315282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8821.658114] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8825.638514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8825.677339] Lustre: 258107:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8825.769843] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8827.030727] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8830.729527] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8832.545401] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8834.788431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8834.882198] Lustre: lustre-OST0000: new disk, initializing [ 8834.883449] Lustre: Skipped 1 previous similar message [ 8834.884909] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8834.886404] Lustre: Skipped 2 previous similar messages [ 8834.888291] Lustre: 260078:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8836.088186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8836.091514] Lustre: Skipped 1 previous similar message [ 8836.093642] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8836.121305] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8836.819494] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8840.570028] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8841.953696] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8842.020602] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8843.442672] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8843.515638] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8846.305028] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8846.305943] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8846.308381] LustreError: Skipped 1 previous similar message [ 8846.309103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8846.313489] Lustre: Skipped 4 previous similar messages [ 8846.315400] Lustre: Skipped 3 previous similar messages [ 8850.323370] Lustre: server umount lustre-OST0000 complete [ 8856.544551] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8856.544748] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8856.550890] Lustre: Skipped 1 previous similar message [ 8857.748930] Lustre: server umount lustre-MDT0000 complete [ 8859.009239] LustreError: 258101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788173287 with bad export cookie 1278875278001127481 [ 8859.012703] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8859.014288] LustreError: 258101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8859.126895] Lustre: server umount lustre-MDT0001 complete [ 8861.878169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8862.010957] LustreError: 262291:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8862.016221] LustreError: 262291:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8862.034370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8862.036555] Lustre: Skipped 2 previous similar messages [ 8863.271545] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8865.896994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8867.299243] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8869.551209] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8871.248384] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8873.347810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8875.390989] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8877.858315] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9023.967725] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9023.970851] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9023.977446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9023.980125] Lustre: Skipped 3 previous similar messages [ 9028.366752] Lustre: server umount lustre-OST0000 complete [ 9035.795351] Lustre: server umount lustre-MDT0000 complete [ 9037.039286] LustreError: 262271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788173465 with bad export cookie 1278875278001128363 [ 9037.043392] LustreError: 262271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9037.044074] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9037.169087] Lustre: server umount lustre-MDT0001 complete [ 9143.636268] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 9144.733597] Key type lgssc unregistered [ 9144.877402] LNet: 265902:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9144.880408] LNetError: 265902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9144.887290] LNet: Removed LNI 192.168.203.145@tcp [ 9145.198103] Key type .llcrypt unregistered [ 9145.199147] Key type ._llcrypt unregistered [ 9152.823319] Key type ._llcrypt registered [ 9152.824535] Key type .llcrypt registered [ 9152.864712] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 9158.949319] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 9159.256196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9159.275048] alg: No test for adler32 (adler32-zlib) [ 9160.125772] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 9160.211814] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 9161.791177] Key type lgssc registered [ 9162.124472] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9176.293583] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 9179.414118] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9179.418856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9180.491526] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9180.499586] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9180.532935] Lustre: lustre-MDT0000: new disk, initializing [ 9180.553595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9180.559046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9181.793884] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9185.708484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9185.748055] Lustre: 269942:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9185.776492] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9185.778764] Lustre: Skipped 1 previous similar message [ 9185.815208] Lustre: lustre-MDT0001: new disk, initializing [ 9185.843399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9185.853880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9185.856880] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9187.052459] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9189.252841] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9191.519277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9191.619747] Lustre: lustre-OST0000: new disk, initializing [ 9191.621435] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9191.624144] Lustre: 271571:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9191.652780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9193.462662] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9194.236971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9194.240037] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9194.273234] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9197.545092] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9197.619532] Lustre: lustre-OST0001: new disk, initializing [ 9197.621665] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9197.625113] Lustre: 272444:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9197.660917] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9199.527112] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9203.702113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9203.705305] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9203.731613] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9204.847512] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9209.468435] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9213.921491] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9213.925540] Lustre: Skipped 2 previous similar messages [ 9213.926860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9213.929152] Lustre: Skipped 3 previous similar messages [ 9218.337695] Lustre: server umount lustre-MDT0000 complete [ 9219.482930] LustreError: 272443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788173648 with bad export cookie 2575521998311178743 [ 9219.485862] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9219.487023] LustreError: 272443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9219.599829] Lustre: server umount lustre-MDT0001 complete [ 9230.344793] Lustre: server umount lustre-OST0000 complete [ 9241.815609] Lustre: server umount lustre-OST0001 complete [ 9246.164536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9249.042448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9251.604073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9254.641996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9260.659749] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 9263.783033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9263.790878] Lustre: lustre-MDT0000: reset Object Index mappings [ 9263.922972] LustreError: 277444:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9263.930392] LustreError: 277444:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 9263.937511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9265.234246] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9267.868739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9267.875629] Lustre: lustre-MDT0001: reset Object Index mappings [ 9269.380864] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9270.298222] Lustre: 278584:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9272.383221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9272.499274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9272.502602] Lustre: Skipped 1 previous similar message [ 9273.510027] LustreError: 278941:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9273.514459] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9274.519452] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9277.363649] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9278.440545] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9279.201456] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9282.180129] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9283.283852] Lustre: 280454:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9288.674873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9288.676880] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9288.679320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9288.680316] Lustre: Skipped 1 previous similar message [ 9290.531250] Lustre: server umount lustre-MDT0000 complete [ 9291.834160] LustreError: 277425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788173720 with bad export cookie 2575521998311185519 [ 9291.837862] LustreError: 277425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9291.837946] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9291.949464] Lustre: server umount lustre-MDT0001 complete [ 9304.001996] Lustre: server umount lustre-OST0000 complete [ 9315.480658] Lustre: server umount lustre-OST0001 complete [ 9321.534365] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 06:55:49 (1788173749) [ 9326.434389] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 9329.323844] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 9340.332714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9340.412820] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9340.423050] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9340.459204] Lustre: lustre-MDT0000: new disk, initializing [ 9340.480546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9340.484064] Lustre: Skipped 1 previous similar message [ 9340.490177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9341.839670] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9345.925499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9345.964840] Lustre: 284700:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9346.062730] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9347.454950] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9351.235118] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9353.139375] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9355.446414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9355.550405] Lustre: lustre-OST0000: new disk, initializing [ 9355.552744] Lustre: Skipped 1 previous similar message [ 9355.555506] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9355.557924] Lustre: Skipped 2 previous similar messages [ 9355.560500] Lustre: 286669:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9356.668150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9356.672327] Lustre: Skipped 1 previous similar message [ 9356.675064] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9356.700160] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9357.509190] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9361.243956] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9362.770196] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9362.843222] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9364.199976] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9364.274871] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9367.010635] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9367.011196] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9367.013369] LustreError: Skipped 1 previous similar message [ 9367.014089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9367.017295] Lustre: Skipped 3 previous similar messages [ 9367.018743] Lustre: Skipped 3 previous similar messages [ 9372.130462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9372.132569] Lustre: Skipped 2 previous similar messages [ 9377.250024] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9377.251954] Lustre: Skipped 1 previous similar message [ 9379.295198] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9379.353283] Lustre: server umount lustre-OST0000 complete [ 9381.855787] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9381.859622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9386.781335] Lustre: server umount lustre-MDT0000 complete [ 9387.488059] LustreError: 285784:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9387.496654] LustreError: 285784:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9388.036258] LustreError: 286668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788173816 with bad export cookie 2575521998311187591 [ 9388.040018] LustreError: 286668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9388.040558] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9388.149936] Lustre: server umount lustre-MDT0001 complete [ 9394.344284] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 9397.971621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9398.090280] LustreError: 289597:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9398.101309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9398.103823] Lustre: Skipped 2 previous similar messages [ 9399.322876] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9402.013604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9403.416341] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9404.333133] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9406.492765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9408.863940] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9411.803210] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9411.862412] Lustre: lustre-OST0001: new disk, initializing [ 9411.864747] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9411.868273] Lustre: 291905:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9413.313904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9413.317289] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9413.349485] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9413.745343] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9418.022386] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9419.094669] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9420.696966] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9420.713057] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9422.815752] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9422.817920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9422.822427] Lustre: Skipped 1 previous similar message [ 9422.823964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9422.825790] Lustre: Skipped 3 previous similar messages [ 9428.442936] Lustre: server umount lustre-MDT0000 complete [ 9428.961617] LustreError: 289594:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9428.966976] LustreError: 289594:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9429.552143] LustreError: 289578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788173858 with bad export cookie 2575521998311188473 [ 9429.556033] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9429.556780] LustreError: 289578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9429.666259] Lustre: server umount lustre-MDT0001 complete [ 9440.264315] Lustre: server umount lustre-OST0000 complete [ 9451.751314] Lustre: server umount lustre-OST0001 complete [ 9457.325125] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 9460.797255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9460.931184] LustreError: 294953:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9462.286207] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9465.077509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9465.216718] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9465.219673] Lustre: Skipped 4 previous similar messages [ 9466.587095] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9467.539906] Lustre: 296092:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9469.726978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9470.887101] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9471.804220] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9474.727554] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9475.825863] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9476.849920] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9479.832333] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9480.942259] Lustre: 297969:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9482.473512] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9482.478031] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9485.368615] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 06:58:33 (1788173913) [ 9501.151707] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9501.154959] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9501.158733] Lustre: Skipped 3 previous similar messages [ 9501.160408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9501.162149] Lustre: Skipped 4 previous similar messages [ 9502.996839] Lustre: server umount lustre-MDT0000 complete [ 9504.099053] LustreError: 294934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788173932 with bad export cookie 2575521998311194780 [ 9504.101859] LustreError: 294934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9504.102405] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9506.784266] LustreError: 296468:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9506.793070] LustreError: 296468:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9510.261632] Lustre: server umount lustre-MDT0001 complete [ 9517.633250] Lustre: server umount lustre-OST0000 complete [ 9525.011902] Lustre: server umount lustre-OST0001 complete [ 9526.982113] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 9530.479057] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 9544.519281] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 9547.599373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9547.673717] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9547.686378] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9547.719253] Lustre: lustre-MDT0000: new disk, initializing [ 9547.742257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9548.946464] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9552.762544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9552.804031] Lustre: 302194:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9552.902230] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9554.102881] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9558.279926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9558.374916] Lustre: 303831:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9560.181962] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9560.350579] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9564.088081] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9565.343972] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9566.044930] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9570.373244] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9575.180346] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 07:00:02 (1788174002) [ 9575.905748] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9575.907806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9575.910467] Lustre: Skipped 8 previous similar messages [ 9575.914945] Lustre: Skipped 10 previous similar messages [ 9581.976614] Lustre: server umount lustre-MDT0000 complete [ 9583.078170] LustreError: 302187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788174011 with bad export cookie 2575521998311202662 [ 9583.081363] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9583.081448] LustreError: 302187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9583.191238] Lustre: server umount lustre-MDT0001 complete [ 9594.368990] Lustre: server umount lustre-OST0000 complete [ 9605.797212] Lustre: server umount lustre-OST0001 complete [ 9608.685479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9608.857806] LustreError: 306942:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9608.863700] LustreError: 306942:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9608.889886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9608.893010] Lustre: Skipped 6 previous similar messages [ 9610.285458] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9613.057565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9614.556059] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9616.887324] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9618.671251] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9620.938848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9623.083470] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9625.583732] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9641.863924] loop1: detected capacity change from 0 to 204800000 [ 9641.951772] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9641.957234] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9641.961953] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9643.719907] loop1: detected capacity change from 0 to 204800000 [ 9645.588103] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9645.618912] LustreError: 306931:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs32201: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 9645.623328] LustreError: 306931:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs32201 [ 9645.625167] LustreError: 306931:0:(mgs_handler.c:517:mgs_target_reg()) fs32201-MDT0001: Error checking target index: rc = -13 [ 9645.628131] LustreError: fs32201-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 9645.630953] LustreError: 310691:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 9645.633208] LustreError: 310691:0:(tgt_mount.c:2129:server_put_super()) no obd fs32201-MDT0001 [ 9645.635069] LustreError: 310691:0:(tgt_mount.c:116:server_deregister_mount()) fs32201-MDT0001 not registered [ 9645.645570] Lustre: server umount fs32201-MDT0001 complete [ 9645.646832] LustreError: 310691:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 9648.445167] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9648.477116] Lustre: Setting parameter fs32201-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs32201-MDT0001 [ 9648.483596] Lustre: Skipped 2 previous similar messages [ 9648.500953] Lustre: srv-fs32201-MDT0001: No data found on store. Initialize space. [ 9648.502951] Lustre: Skipped 4 previous similar messages [ 9648.538726] Lustre: fs32201-MDT0001: new disk, initializing [ 9648.540473] Lustre: Skipped 3 previous similar messages [ 9648.577387] Lustre: srv-fs32201-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 9648.579737] Lustre: 311157:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs32201-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9648.582773] LustreError: 311157:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs32201-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9649.993675] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9652.879231] Lustre: server umount fs32201-MDT0001 complete [ 9657.297424] loop1: detected capacity change from 0 to 204800000 [ 9659.287050] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9659.390849] Lustre: 312721:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs32201-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9659.393206] LustreError: 312721:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs32201-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9660.590945] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9662.025746] Lustre: server umount fs32201-MDT0001 complete [ 9666.605448] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.45@tcp (stopping) [ 9666.607487] Lustre: Skipped 4 previous similar messages [ 9669.601905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9669.603429] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9671.264745] Lustre: server umount lustre-MDT0000 complete [ 9674.160036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9674.205916] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9674.276140] LustreError: 309455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9674.282309] LustreError: 309455:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 26 previous similar messages [ 9675.724329] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9678.317109] loop1: detected capacity change from 0 to 204800000 [ 9679.335466] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9680.258128] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9680.385196] Lustre: 315165:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs32201-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9680.387992] LustreError: 315165:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs32201-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9681.737278] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9683.372426] Lustre: server umount fs32201-MDT0001 complete [ 9690.079983] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9696.026421] Lustre: server umount lustre-OST0000 complete [ 9703.516248] Lustre: server umount lustre-MDT0000 complete [ 9704.808147] LustreError: 313212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788174133 with bad export cookie 2575521998311205336 [ 9704.813836] LustreError: 313212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9704.927220] Lustre: server umount lustre-MDT0001 complete [ 9708.641498] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 9709.703965] Key type lgssc unregistered [ 9709.833460] LNet: 317505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9709.837492] LNetError: 317505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9709.844389] LNet: Removed LNI 192.168.203.145@tcp [ 9710.187107] Key type .llcrypt unregistered [ 9710.188996] Key type ._llcrypt unregistered [ 9717.968293] Key type ._llcrypt registered [ 9717.969876] Key type .llcrypt registered [ 9718.007030] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 9718.326349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9718.366358] alg: No test for adler32 (adler32-zlib) [ 9719.233881] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 9719.322375] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 9720.911162] Key type lgssc registered [ 9721.231474] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9723.576284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9723.581679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9724.700484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9725.926038] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9728.470238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9728.597945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9729.859879] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9732.031518] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9733.788542] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9736.167186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9736.299197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9738.342250] LustreError: 320582:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9738.348940] LustreError: 320582:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9738.361484] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9740.784749] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9743.841853] LustreError: 320581:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9748.965371] LustreError: 320582:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9751.590856] LustreError: 320940:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9751.595186] LustreError: 320940:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9756.706238] LustreError: 320583:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9756.713025] LustreError: 320583:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9757.631611] loop1: detected capacity change from 0 to 204800000 [ 9757.701234] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9757.705587] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9757.708793] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9759.360052] loop1: detected capacity change from 0 to 204800000 [ 9761.096450] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9761.140225] Lustre: fs2089-OST0000: new disk, initializing [ 9761.142512] Lustre: srv-fs2089-OST0000: No data found on store. Initialize space. [ 9761.144769] Lustre: 322506:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9761.164793] Lustre: fs2089-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9762.967780] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9765.351085] LustreError: 320940:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9765.355895] LustreError: 320940:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9766.215666] Lustre: server umount fs2089-OST0000 complete [ 9770.275860] loop1: detected capacity change from 0 to 204800000 [ 9772.128873] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9772.190700] Lustre: fs2089-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9774.101855] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9777.020103] loop2: detected capacity change from 0 to 204800000 [ 9777.120933] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9777.125924] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9777.131099] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9778.871396] loop2: detected capacity change from 0 to 204800000 [ 9780.783039] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9780.815042] LustreError: fs2089-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 9780.818882] LustreError: fs2089-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 9780.822123] LustreError: 325542:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 9780.824474] LustreError: 325542:0:(tgt_mount.c:2129:server_put_super()) no obd fs2089-OST0001 [ 9780.826444] LustreError: 325542:0:(tgt_mount.c:116:server_deregister_mount()) fs2089-OST0001 not registered [ 9780.838741] Lustre: server umount fs2089-OST0001 complete [ 9780.840653] LustreError: 325542:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 9782.305951] LustreError: 321581:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9782.313345] LustreError: 321581:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 9784.400774] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9784.448108] Lustre: fs2089-OST0001: new disk, initializing [ 9784.450254] Lustre: srv-fs2089-OST0001: No data found on store. Initialize space. [ 9784.453312] Lustre: 326092:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9784.479781] Lustre: fs2089-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9786.536470] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9789.983389] Lustre: server umount fs2089-OST0000 complete [ 9793.048939] Lustre: server umount fs2089-OST0001 complete [ 9797.223488] loop1: detected capacity change from 0 to 204800000 [ 9799.188818] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9801.302669] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9803.611085] loop2: detected capacity change from 0 to 204800000 [ 9805.534385] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9805.597157] Lustre: fs2089-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9805.599604] Lustre: Skipped 1 previous similar message [ 9807.694403] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9810.825308] Lustre: server umount fs2089-OST0000 complete [ 9813.975419] Lustre: server umount fs2089-OST0001 complete [ 9816.546807] LustreError: 320582:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9816.553697] LustreError: 320582:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [ 9820.640601] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9820.644599] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9820.649503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9821.664571] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9821.668291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9821.668327] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9823.829842] Lustre: server umount lustre-OST0000 complete [ 9825.759746] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9825.761944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9825.765467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9825.767070] Lustre: Skipped 1 previous similar message [ 9831.259889] Lustre: server umount lustre-MDT0000 complete [ 9832.545157] LustreError: 318735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788174261 with bad export cookie 10395850012673464675 [ 9832.549055] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9832.551207] LustreError: 318735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9832.671167] Lustre: server umount lustre-MDT0001 complete [ 9836.454052] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 9837.557501] Key type lgssc unregistered [ 9837.687348] LNet: 332007:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9837.690171] LNetError: 332007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9837.697333] LNet: Removed LNI 192.168.203.145@tcp [ 9838.008130] Key type .llcrypt unregistered [ 9838.009791] Key type ._llcrypt unregistered [ 9848.154875] Key type ._llcrypt registered [ 9848.157209] Key type .llcrypt registered [ 9848.191548] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 9854.148477] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 9854.550286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9854.614079] alg: No test for adler32 (adler32-zlib) [ 9855.450397] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 9855.537914] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 9857.127169] Key type lgssc registered [ 9857.486590] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9869.371852] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9869.379626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9870.472455] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9870.484596] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9870.517441] Lustre: lustre-MDT0000: new disk, initializing [ 9870.541237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9870.547758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9871.913159] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9876.061223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9876.104060] Lustre: 335837:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9876.131237] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9876.133968] Lustre: Skipped 1 previous similar message [ 9876.177441] Lustre: lustre-MDT0001: new disk, initializing [ 9876.219354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9876.231675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9876.235280] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9877.576571] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9881.210220] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9883.000642] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9885.180759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9885.282880] Lustre: lustre-OST0000: new disk, initializing [ 9885.284462] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9885.286820] Lustre: 337804:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9885.313527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9887.115978] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9887.157573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9887.162513] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9887.202148] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9890.590908] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9891.227615] Lustre: Mounted lustre-client - version 2.17.56_54_gb664824 [ 9891.548779] Lustre: Unmounted lustre-client [ 9896.282264] Lustre: server umount lustre-OST0000 complete [ 9901.536693] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9901.538161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9901.543295] Lustre: Skipped 1 previous similar message [ 9906.656030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9906.660138] Lustre: Skipped 2 previous similar messages [ 9907.705956] Lustre: server umount lustre-MDT0000 complete [ 9908.928194] LustreError: 336633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788174337 with bad export cookie 5219860094016204567 [ 9908.930750] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9908.934709] LustreError: 336633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9918.482988] Lustre: server umount lustre-MDT0001 complete [ 9922.024119] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 9923.065594] Key type lgssc unregistered [ 9947.743395] LNet: 339965:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9947.746724] LNetError: 339965:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10020.453307] LNet: Removed LNI 192.168.203.145@tcp [10020.739693] Key type .llcrypt unregistered [10020.741374] Key type ._llcrypt unregistered [10026.505118] Key type ._llcrypt registered [10026.506152] Key type .llcrypt registered [10026.539766] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [10032.728146] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [10033.223034] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10033.232365] alg: No test for adler32 (adler32-zlib) [10034.081616] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [10034.164816] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [10035.743219] Key type lgssc registered [10036.165860] Lustre: Echo OBD driver; http://www.lustre.org/ [10046.473478] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 07:07:54 (1788174474) [10048.570808] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10048.575630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10049.653574] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10049.662479] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10049.694988] Lustre: lustre-MDT0000: new disk, initializing [10049.714386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10049.719970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10050.984570] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10054.962662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10054.999229] Lustre: 343165:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10055.018863] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10055.020428] Lustre: Skipped 1 previous similar message [10055.053568] Lustre: lustre-MDT0001: new disk, initializing [10055.085674] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10055.098472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10055.101684] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10056.320805] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10059.719780] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10061.358370] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10063.414728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10063.536829] Lustre: lustre-OST0000: new disk, initializing [10063.539078] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10063.541751] Lustre: 345131:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10063.575451] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10063.825120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10063.828146] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10063.858923] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10065.303554] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10068.799583] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10079.200438] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10079.203155] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10079.206791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10079.208496] Lustre: Skipped 1 previous similar message [10080.463609] Lustre: server umount lustre-OST0000 complete [10084.320618] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10084.321144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10084.325527] Lustre: Skipped 2 previous similar messages [10087.699106] Lustre: server umount lustre-MDT0000 complete [10088.828312] LustreError: 343156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788174517 with bad export cookie 8040075507838705393 [10088.830619] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10088.831924] LustreError: 343156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10088.941822] Lustre: server umount lustre-MDT0001 complete [10092.568715] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [10093.513128] Key type lgssc unregistered [10093.634313] LNet: 347041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10093.636749] LNetError: 347041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10093.643289] LNet: Removed LNI 192.168.203.145@tcp [10093.947081] Key type .llcrypt unregistered [10093.948702] Key type ._llcrypt unregistered [10101.712729] Key type ._llcrypt registered [10101.714701] Key type .llcrypt registered [10101.747130] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [10102.095322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10102.143140] alg: No test for adler32 (adler32-zlib) [10102.983101] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [10103.063666] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [10104.631147] Key type lgssc registered [10104.963319] Lustre: Echo OBD driver; http://www.lustre.org/ [10107.474461] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10107.480141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10108.600666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10109.844877] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10112.522043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10112.637088] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10113.948524] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10116.253883] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10117.978112] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10120.023817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10120.131592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10122.016667] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10124.241691] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10137.668788] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10138.170119] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10139.617052] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10139.618104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10139.621826] Lustre: Skipped 2 previous similar messages [10139.624093] Lustre: Skipped 1 previous similar message [10141.658493] Lustre: server umount lustre-MDT0000 complete [10142.871350] LustreError: 348273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788174571 with bad export cookie 3303617895673541421 [10142.875924] LustreError: 348273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10142.877237] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10144.735840] LustreError: 350494:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10144.736465] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10144.743492] LustreError: 350494:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10144.750233] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10144.755494] Lustre: Skipped 2 previous similar messages [10145.003516] Lustre: server umount lustre-MDT0001 complete [10149.927107] Lustre: lustre-OST0000: Not available for connect from 192.168.203.45@tcp (stopping) [10149.929394] Lustre: Skipped 1 previous similar message [10152.334217] Lustre: server umount lustre-OST0000 complete [10155.224972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10156.537432] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10157.424724] Lustre: 352483:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.145@tcp [10158.121881] Lustre: server umount MGS complete [10160.908343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10161.049805] LustreError: 353053:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10161.056179] LustreError: 353053:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10161.074186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10162.383734] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10165.170171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10165.280020] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10166.622848] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10169.507432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10176.545385] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10178.973808] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10179.517265] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [10181.274119] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10181.843058] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10186.209306] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10186.211920] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10186.213230] LustreError: Skipped 1 previous similar message [10186.213823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10186.218225] Lustre: Skipped 1 previous similar message [10186.223713] Lustre: Skipped 1 previous similar message [10189.004613] Lustre: server umount lustre-OST0000 complete [10190.815848] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10190.818799] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10190.823358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10196.441017] Lustre: server umount lustre-MDT0000 complete [10196.447918] LustreError: 353053:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10196.455364] LustreError: 353053:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10197.690229] LustreError: 353034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788174626 with bad export cookie 3303617895673542247 [10197.693965] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10197.694659] LustreError: 353034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10197.820350] Lustre: server umount lustre-MDT0001 complete [10201.304173] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [10202.278739] Key type lgssc unregistered [10202.399327] LNet: 356490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10202.401772] LNetError: 356490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10202.408280] LNet: Removed LNI 192.168.203.145@tcp [10202.704821] Key type .llcrypt unregistered [10202.706241] Key type ._llcrypt unregistered [10208.687335] Key type ._llcrypt registered [10208.688159] Key type .llcrypt registered [10208.721932] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [10214.771075] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [10215.022061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10215.028829] alg: No test for adler32 (adler32-zlib) [10215.875325] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [10215.957618] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [10217.527181] Key type lgssc registered [10217.885763] Lustre: Echo OBD driver; http://www.lustre.org/ [10229.850459] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10229.858946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10230.948335] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10230.961055] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10230.996877] Lustre: lustre-MDT0000: new disk, initializing [10231.024842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10231.033769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10232.520173] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10237.063602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10237.110209] Lustre: 359623:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10237.135651] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10237.137767] Lustre: Skipped 1 previous similar message [10237.172085] Lustre: lustre-MDT0001: new disk, initializing [10237.210308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10237.226745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10237.231050] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10238.729691] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10242.669112] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10244.612940] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10247.047247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10247.158944] Lustre: lustre-OST0000: new disk, initializing [10247.160658] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10247.164431] Lustre: 361589:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10247.202559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10249.247473] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10252.951474] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10254.377747] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10254.841027] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10254.845655] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10254.878770] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10255.469546] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [10256.870863] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10256.951056] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10259.937980] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10259.938394] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10259.941302] LustreError: Skipped 1 previous similar message [10259.946122] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10259.949254] Lustre: Skipped 1 previous similar message [10263.760338] Lustre: server umount lustre-OST0000 complete [10265.057562] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10265.058198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10265.061047] Lustre: Skipped 2 previous similar messages [10265.064079] Lustre: Skipped 1 previous similar message [10270.175927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10270.177739] Lustre: Skipped 1 previous similar message [10275.296517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10275.298331] Lustre: Skipped 1 previous similar message [10279.391133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10279.393540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10279.397299] Lustre: Skipped 1 previous similar message [10279.462368] Lustre: server umount lustre-MDT0000 complete [10280.759381] LustreError: 359616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788174709 with bad export cookie 16066679407149395578 [10280.762536] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10280.763109] LustreError: 359616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10280.888480] Lustre: server umount lustre-MDT0001 complete [10283.932511] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 07:11:51 (1788174711) [10289.647749] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [10293.216127] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [10306.455828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10306.555214] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10306.568346] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10306.606863] Lustre: lustre-MDT0000: new disk, initializing [10306.630138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10306.638242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10308.174996] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10312.521717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10312.572674] Lustre: 366347:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10312.599413] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10312.601386] Lustre: Skipped 1 previous similar message [10312.634450] Lustre: lustre-MDT0001: new disk, initializing [10312.669125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10312.683523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10312.688259] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10314.059925] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10317.699469] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10319.567864] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10321.804324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10321.905170] Lustre: lustre-OST0000: new disk, initializing [10321.906670] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10321.910055] Lustre: 368313:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10321.938314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10323.901537] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10327.032590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10327.036792] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10327.063773] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10327.587102] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10337.249015] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10337.249292] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10337.252535] LustreError: Skipped 1 previous similar message [10337.259081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10337.264094] Lustre: Skipped 1 previous similar message [10339.162407] Lustre: server umount lustre-OST0000 complete [10342.368924] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10342.374574] Lustre: Skipped 2 previous similar messages [10346.653222] Lustre: server umount lustre-MDT0000 complete [10347.973465] LustreError: 366337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788174776 with bad export cookie 16066679407149396446 [10347.976585] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10347.978870] LustreError: 366337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10348.122276] Lustre: server umount lustre-MDT0001 complete [10352.054509] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [10353.224289] Key type lgssc unregistered [10353.362435] LNet: 370465:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10353.365728] LNetError: 370465:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10353.377305] LNet: Removed LNI 192.168.203.145@tcp [10353.713237] Key type .llcrypt unregistered [10353.714347] Key type ._llcrypt unregistered [10388.355267] Key type ._llcrypt registered [10388.356322] Key type .llcrypt registered [10388.394884] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [10388.523068] Key type .llcrypt unregistered [10388.524807] Key type ._llcrypt unregistered [10418.046514] Key type ._llcrypt registered [10418.048276] Key type .llcrypt registered [10418.092719] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [10418.229303] Key type .llcrypt unregistered [10418.231081] Key type ._llcrypt unregistered [10430.797683] Key type ._llcrypt registered [10430.799096] Key type .llcrypt registered [10430.852235] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [10431.006642] Key type .llcrypt unregistered [10431.008177] Key type ._llcrypt unregistered [10443.088082] Key type ._llcrypt registered [10443.089594] Key type .llcrypt registered [10443.134544] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [10443.269422] Key type .llcrypt unregistered [10443.271221] Key type ._llcrypt unregistered [10472.768930] Key type ._llcrypt registered [10472.769909] Key type .llcrypt registered [10472.802469] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [10472.911442] Key type .llcrypt unregistered [10472.913108] Key type ._llcrypt unregistered [10483.955090] Key type ._llcrypt registered [10483.956377] Key type .llcrypt registered [10483.991068] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [10484.087358] Key type .llcrypt unregistered [10484.088703] Key type ._llcrypt unregistered [10495.568532] Key type ._llcrypt registered [10495.569928] Key type .llcrypt registered [10495.614103] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [10495.756556] Key type .llcrypt unregistered [10495.758127] Key type ._llcrypt unregistered [10507.870389] Key type ._llcrypt registered [10507.872063] Key type .llcrypt registered [10507.921599] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [10508.059311] Key type .llcrypt unregistered [10508.060951] Key type ._llcrypt unregistered [10519.558946] Key type ._llcrypt registered [10519.560048] Key type .llcrypt registered [10519.602051] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [10519.723812] Key type .llcrypt unregistered [10519.725470] Key type ._llcrypt unregistered [10530.677493] Key type ._llcrypt registered [10530.679011] Key type .llcrypt registered [10530.721921] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [10530.846631] Key type .llcrypt unregistered [10530.847914] Key type ._llcrypt unregistered [10551.519460] Key type ._llcrypt registered [10551.520301] Key type .llcrypt registered [10551.560532] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [10551.666270] Key type .llcrypt unregistered [10551.668505] Key type ._llcrypt unregistered [10562.902150] Key type ._llcrypt registered [10562.904120] Key type .llcrypt registered [10562.955411] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [10563.070366] Key type .llcrypt unregistered [10563.072134] Key type ._llcrypt unregistered [10575.625239] Key type ._llcrypt registered [10575.626522] Key type .llcrypt registered [10575.672229] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [10575.813506] Key type .llcrypt unregistered [10575.815180] Key type ._llcrypt unregistered [10594.121142] Key type ._llcrypt registered [10594.122126] Key type .llcrypt registered [10594.158552] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [10600.692988] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [10601.028593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10601.048058] alg: No test for adler32 (adler32-zlib) [10601.909497] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [10602.009835] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [10603.599208] Key type lgssc registered [10603.984086] Lustre: Echo OBD driver; http://www.lustre.org/ [10615.446678] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10615.451786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10616.533549] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [10616.546157] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [10616.585198] Lustre: tst250fs-MDT0000: new disk, initializing [10616.611951] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10616.620386] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10618.058082] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10622.306681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10622.351356] Lustre: 390431:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10622.385536] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [10622.387950] Lustre: Skipped 1 previous similar message [10622.423983] Lustre: tst250fs-MDT0001: new disk, initializing [10622.457099] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10622.471655] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10622.476183] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10623.935298] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10626.374370] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10628.702796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10628.809328] Lustre: tst250fs-OST0000: new disk, initializing [10628.811544] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [10628.815406] Lustre: 392062:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10628.855095] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10630.966622] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10631.612359] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10631.615784] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10631.658322] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10635.294136] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10635.364157] Lustre: tst250fs-OST0001: new disk, initializing [10635.366812] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [10635.370373] Lustre: 392936:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10635.405742] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10637.477439] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10637.899332] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10637.904579] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10637.930056] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10640.086219] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [10653.151873] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10653.155118] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10653.161146] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [10653.665725] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10653.669072] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [10653.671922] Lustre: Skipped 1 previous similar message [10653.677304] Lustre: Skipped 2 previous similar messages [10656.412528] Lustre: server umount tst250fs-MDT0000 complete [10657.689385] LustreError: 390425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788175086 with bad export cookie 11819452609169986454 [10657.693635] LustreError: 390425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10657.694518] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10657.817538] Lustre: server umount tst250fs-MDT0001 complete [10669.381625] Lustre: server umount tst250fs-OST0000 complete [10681.072703] Lustre: server umount tst250fs-OST0001 complete [10687.237693] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [10690.019409] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [10700.457115] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 07:18:48 (1788175128) [10701.047843] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [10706.394133] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [10709.928811] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [10720.353119] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10479 sec ========= 07:19:08 (1788175148) [10720.881695] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 07:19:08 (1788175148) === [10722.014757] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 07:19:09 (1788175149) === [10735.506542] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [10736.617411] Key type lgssc unregistered [10736.748384] LNet: 402049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10736.752140] LNetError: 402049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10736.762267] LNet: Removed LNI 192.168.203.145@tcp [10737.064119] Key type .llcrypt unregistered [10737.066524] Key type ._llcrypt unregistered